2026-02-21T01:09:26.568 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-21T01:09:26.583 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-21T01:09:26.668 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-21_00:21:18-rados-wip-yuri3-testing-2026-02-19-1610-tentacle-distro-default-trial/62135 branch: wip-yuri3-testing-2026-02-19-1610-tentacle description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '62135' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-21_00:21:18-rados-wip-yuri3-testing-2026-02-19-1610-tentacle-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri3-testing-2026-02-19-1610-tentacle ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 36d388f27052dd940bdbaf543bb05bdba0acc155 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 36d388f27052dd940bdbaf543bb05bdba0acc155 workunit: branch: wip-yuri3-testing-2026-02-19-1610-tentacle sha1: 36d388f27052dd940bdbaf543bb05bdba0acc155 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6877 sha1: 36d388f27052dd940bdbaf543bb05bdba0acc155 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri3-testing-2026-02-19-1610-tentacle suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_36d388f27052dd940bdbaf543bb05bdba0acc155/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 36d388f27052dd940bdbaf543bb05bdba0acc155 targets: trial106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJYMm+p7+QBfoRl5OBmLB/KESaic7HjKgX/DMjY9VeApmSZfN+30HlpuZo5FnYazz0XusY9QLH+vBD2oZXlUGT0= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-21_00:21:18 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-21T01:09:26.668 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_36d388f27052dd940bdbaf543bb05bdba0acc155/qa; will attempt to use it 2026-02-21T01:09:26.669 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_36d388f27052dd940bdbaf543bb05bdba0acc155/qa/tasks 2026-02-21T01:09:26.669 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-21T01:09:26.671 INFO:teuthology.task.internal:Checking packages... 2026-02-21T01:09:26.671 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '36d388f27052dd940bdbaf543bb05bdba0acc155' 2026-02-21T01:09:26.671 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-21T01:09:26.671 INFO:teuthology.packaging:ref: None 2026-02-21T01:09:26.671 INFO:teuthology.packaging:tag: None 2026-02-21T01:09:26.671 INFO:teuthology.packaging:branch: wip-yuri3-testing-2026-02-19-1610-tentacle 2026-02-21T01:09:26.671 INFO:teuthology.packaging:sha1: 36d388f27052dd940bdbaf543bb05bdba0acc155 2026-02-21T01:09:26.671 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri3-testing-2026-02-19-1610-tentacle 2026-02-21T01:09:26.795 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-681-g36d388f2-1jammy 2026-02-21T01:09:26.796 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-21T01:09:26.810 INFO:teuthology.task.internal:no buildpackages task found 2026-02-21T01:09:26.810 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-21T01:09:26.824 INFO:teuthology.task.internal:Saving configuration 2026-02-21T01:09:26.833 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-21T01:09:26.845 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-21T01:09:26.928 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial106.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-21_00:21:18-rados-wip-yuri3-testing-2026-02-19-1610-tentacle-distro-default-trial/62135', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-21 01:05:24.940506', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHPugWHyWkN6OJE+NScExzWkEiPUHCbJP+MYXccg0Wui'} 2026-02-21T01:09:26.928 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-21T01:09:26.941 INFO:teuthology.task.internal:roles: ubuntu@trial106.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-21T01:09:26.941 INFO:teuthology.run_tasks:Running task console_log... 2026-02-21T01:09:27.028 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7044483ff7e0>, signals=[15]) 2026-02-21T01:09:27.028 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-21T01:09:27.042 INFO:teuthology.task.internal:Opening connections... 2026-02-21T01:09:27.042 DEBUG:teuthology.task.internal:connecting to ubuntu@trial106.front.sepia.ceph.com 2026-02-21T01:09:27.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-21T01:09:27.106 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-21T01:09:27.118 DEBUG:teuthology.orchestra.run.trial106:> uname -m 2026-02-21T01:09:27.187 INFO:teuthology.orchestra.run.trial106.stdout:x86_64 2026-02-21T01:09:27.187 DEBUG:teuthology.orchestra.run.trial106:> cat /etc/os-release 2026-02-21T01:09:27.232 INFO:teuthology.orchestra.run.trial106.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-21T01:09:27.232 INFO:teuthology.orchestra.run.trial106.stdout:NAME="Ubuntu" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:VERSION_ID="22.04" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:VERSION_CODENAME=jammy 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:ID=ubuntu 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:ID_LIKE=debian 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-21T01:09:27.233 INFO:teuthology.orchestra.run.trial106.stdout:UBUNTU_CODENAME=jammy 2026-02-21T01:09:27.233 INFO:teuthology.lock.ops:Updating trial106.front.sepia.ceph.com on lock server 2026-02-21T01:09:27.314 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-21T01:09:27.327 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-21T01:09:27.340 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-21T01:09:27.340 DEBUG:teuthology.orchestra.run.trial106:> test '!' -e /home/ubuntu/cephtest 2026-02-21T01:09:27.344 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-21T01:09:27.354 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-21T01:09:27.354 DEBUG:teuthology.orchestra.run.trial106:> test -z $(ls -A /var/lib/ceph) 2026-02-21T01:09:27.389 INFO:teuthology.orchestra.run.trial106.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-21T01:09:27.390 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-21T01:09:27.479 INFO:teuthology.run_tasks:Running task kernel... 2026-02-21T01:09:27.498 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-21T01:09:27.499 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-21T01:09:27.499 DEBUG:teuthology.orchestra.run.trial106:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-21T01:09:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T01:09:27.502 DEBUG:teuthology.orchestra.run.trial106:> uname -r 2026-02-21T01:09:27.552 INFO:teuthology.orchestra.run.trial106.stdout:5.15.0-170-generic 2026-02-21T01:09:27.552 INFO:teuthology.task.kernel:Running kernel on trial106: 5.15.0-170-generic 2026-02-21T01:09:27.552 DEBUG:teuthology.orchestra.run.trial106:> sudo apt-get clean 2026-02-21T01:09:27.611 DEBUG:teuthology.orchestra.run.trial106:> sudo apt-get update 2026-02-21T01:09:28.128 INFO:teuthology.orchestra.run.trial106.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-21T01:09:28.201 INFO:teuthology.orchestra.run.trial106.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-21T01:09:28.477 INFO:teuthology.orchestra.run.trial106.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-21T01:09:28.599 INFO:teuthology.orchestra.run.trial106.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-21T01:09:28.713 INFO:teuthology.orchestra.run.trial106.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-21T01:09:29.040 INFO:teuthology.orchestra.run.trial106.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-21T01:09:29.049 INFO:teuthology.orchestra.run.trial106.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-21T01:09:29.049 INFO:teuthology.orchestra.run.trial106.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-21T01:09:29.079 INFO:teuthology.orchestra.run.trial106.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-21T01:09:29.084 INFO:teuthology.orchestra.run.trial106.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-21T01:09:29.085 INFO:teuthology.orchestra.run.trial106.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-21T01:09:29.214 INFO:teuthology.orchestra.run.trial106.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-21T01:09:29.232 INFO:teuthology.orchestra.run.trial106.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-21T01:09:29.303 INFO:teuthology.orchestra.run.trial106.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-21T01:09:29.310 INFO:teuthology.orchestra.run.trial106.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-21T01:09:29.311 INFO:teuthology.orchestra.run.trial106.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-21T01:09:29.338 INFO:teuthology.orchestra.run.trial106.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-21T01:09:29.338 INFO:teuthology.orchestra.run.trial106.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-21T01:09:29.460 INFO:teuthology.orchestra.run.trial106.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-21T01:09:33.020 INFO:teuthology.orchestra.run.trial106.stdout:Fetched 22.6 MB in 3s (8339 kB/s) 2026-02-21T01:09:33.532 INFO:teuthology.orchestra.run.trial106.stdout:Reading package lists... 2026-02-21T01:09:33.544 DEBUG:teuthology.orchestra.run.trial106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-21T01:09:33.578 INFO:teuthology.orchestra.run.trial106.stdout:Reading package lists... 2026-02-21T01:09:33.683 INFO:teuthology.orchestra.run.trial106.stdout:Building dependency tree... 2026-02-21T01:09:33.684 INFO:teuthology.orchestra.run.trial106.stdout:Reading state information... 2026-02-21T01:09:33.769 INFO:teuthology.orchestra.run.trial106.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-21T01:09:33.769 INFO:teuthology.orchestra.run.trial106.stdout:linux-image-generic set to manually installed. 2026-02-21T01:09:33.770 INFO:teuthology.orchestra.run.trial106.stdout:The following packages were automatically installed and are no longer required: 2026-02-21T01:09:33.770 INFO:teuthology.orchestra.run.trial106.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-21T01:09:33.770 INFO:teuthology.orchestra.run.trial106.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-21T01:09:33.770 INFO:teuthology.orchestra.run.trial106.stdout: sg3-utils-udev 2026-02-21T01:09:33.770 INFO:teuthology.orchestra.run.trial106.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-21T01:09:33.788 INFO:teuthology.orchestra.run.trial106.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-21T01:09:33.790 DEBUG:teuthology.orchestra.run.trial106:> dpkg -s linux-image-generic 2026-02-21T01:09:33.846 INFO:teuthology.orchestra.run.trial106.stdout:Package: linux-image-generic 2026-02-21T01:09:33.846 INFO:teuthology.orchestra.run.trial106.stdout:Status: install ok installed 2026-02-21T01:09:33.846 INFO:teuthology.orchestra.run.trial106.stdout:Priority: optional 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Section: kernel 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Installed-Size: 21 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Maintainer: Ubuntu Kernel Team 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Architecture: amd64 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Source: linux-meta 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Version: 5.15.0.170.159 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Recommends: thermald 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout:Description: Generic Linux kernel image 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout: This package will always depend on the latest generic kernel image 2026-02-21T01:09:33.847 INFO:teuthology.orchestra.run.trial106.stdout: available. 2026-02-21T01:09:33.847 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-21T01:09:33.847 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-21T01:09:33.847 DEBUG:teuthology.orchestra.run.trial106:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-21T01:09:33.897 INFO:teuthology.orchestra.run.trial106.stdout:ttyS1 2026-02-21T01:09:33.919 DEBUG:teuthology.parallel:result is None 2026-02-21T01:09:33.920 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-21T01:09:33.932 INFO:teuthology.task.internal:Creating test directory... 2026-02-21T01:09:33.932 DEBUG:teuthology.orchestra.run.trial106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-21T01:09:33.969 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-21T01:09:33.983 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-21T01:09:33.996 INFO:teuthology.task.internal:Creating archive directory... 2026-02-21T01:09:33.996 DEBUG:teuthology.orchestra.run.trial106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-21T01:09:34.018 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-21T01:09:34.029 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-21T01:09:34.029 DEBUG:teuthology.orchestra.run.trial106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-21T01:09:34.066 INFO:teuthology.orchestra.run.trial106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:09:34.072 INFO:teuthology.orchestra.run.trial106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:09:34.074 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-21T01:09:34.085 INFO:teuthology.task.internal:Configuring sudo... 2026-02-21T01:09:34.085 DEBUG:teuthology.orchestra.run.trial106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-21T01:09:34.125 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-21T01:09:34.137 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-21T01:09:34.137 DEBUG:teuthology.orchestra.run.trial106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-21T01:09:34.169 DEBUG:teuthology.orchestra.run.trial106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T01:09:34.213 DEBUG:teuthology.orchestra.run.trial106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T01:09:34.261 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:09:34.261 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-21T01:09:34.355 DEBUG:teuthology.orchestra.run.trial106:> sudo service rsyslog restart 2026-02-21T01:09:34.415 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-21T01:09:34.429 INFO:teuthology.task.internal:Starting timer... 2026-02-21T01:09:34.429 INFO:teuthology.run_tasks:Running task pcp... 2026-02-21T01:09:34.449 INFO:teuthology.run_tasks:Running task selinux... 2026-02-21T01:09:34.468 DEBUG:teuthology.task.selinux:Excluding trial106: OS 'ubuntu' does not support SELinux 2026-02-21T01:09:34.468 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-21T01:09:34.468 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-21T01:09:34.468 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-21T01:09:34.469 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-21T01:09:34.483 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-21T01:09:34.487 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-21T01:09:34.602 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-21T01:09:34.612 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-21T01:09:34.614 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial106.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-21T01:11:55.244 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial106.front.sepia.ceph.com')] 2026-02-21T01:11:55.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-21T01:11:55.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-21T01:11:55.329 DEBUG:teuthology.orchestra.run.trial106:> true 2026-02-21T01:11:55.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial106.front.sepia.ceph.com' 2026-02-21T01:11:55.584 INFO:teuthology.run_tasks:Running task clock... 2026-02-21T01:11:55.601 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-21T01:11:55.601 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-21T01:11:55.602 DEBUG:teuthology.orchestra.run.trial106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Command line: ntpd -gq 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: ---------------------------------------------------- 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: ntp-4 is maintained by Network Time Foundation, 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: corporation. Support and training for ntp-4 are 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: available at https://www.nwtime.org/support 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: ---------------------------------------------------- 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: proto: precision = 0.030 usec (-25) 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: basedate set to 2022-02-04 2026-02-21T01:11:55.645 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: gps base set to 2022-02-06 (week 2196) 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: restrict ::: KOD does nothing without LIMITED. 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listen and drop on 0 v6wildcard [::]:123 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listen normally on 2 lo 127.0.0.1:123 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listen normally on 3 enp1s0f0 10.20.193.106:123 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listen normally on 4 lo [::1]:123 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:630a%2]:123 2026-02-21T01:11:55.646 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:11:55 ntpd[9425]: Listening on routing socket on fd #22 for interface updates 2026-02-21T01:11:56.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:56.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:56.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:56.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:56.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:56.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:56.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:11:56.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:11:56.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:56.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:56.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:11:56.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:56 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:11:58.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.648 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.648 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.648 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.648 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.648 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.674 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.674 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:11:58.674 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:11:58.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:11:58.679 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:11:58.708 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:11:58.708 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:11:58.708 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:11:58.708 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:11:58 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:00.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:00.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:00.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:00.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:00.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:00.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:00.708 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:00.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:00.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:00.709 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:00 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:02.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.646 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.647 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:02.673 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:02.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:02.675 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/rawstats.20260221: Permission denied 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/peerstats.20260221: Permission denied 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stdout:21 Feb 01:12:02 ntpd[9425]: ntpd: time set +0.528550 s 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stdout:ntpd: time set +0.528550s 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-21T01:12:02.678 INFO:teuthology.orchestra.run.trial106.stderr:21 Feb 01:12:02 ntpd[9425]: can't open /var/log/ntpstats/loopstats.20260221: Permission denied 2026-02-21T01:12:03.162 INFO:teuthology.orchestra.run.trial106.stdout: remote refid st t when poll reach delay offset jitter 2026-02-21T01:12:03.162 INFO:teuthology.orchestra.run.trial106.stdout:============================================================================== 2026-02-21T01:12:03.163 INFO:teuthology.orchestra.run.trial106.stdout: time1.weber.edu .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-21T01:12:03.163 INFO:teuthology.orchestra.run.trial106.stdout: 163.123.153.14 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-21T01:12:03.163 INFO:teuthology.orchestra.run.trial106.stdout: ntp.maxhost.io .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-21T01:12:03.163 INFO:teuthology.orchestra.run.trial106.stdout: startkeylogger. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-21T01:12:03.163 INFO:teuthology.run_tasks:Running task install... 2026-02-21T01:12:03.175 DEBUG:teuthology.task.install:project ceph 2026-02-21T01:12:03.175 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '36d388f27052dd940bdbaf543bb05bdba0acc155'}} 2026-02-21T01:12:03.175 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '36d388f27052dd940bdbaf543bb05bdba0acc155'} 2026-02-21T01:12:03.175 INFO:teuthology.task.install:Using flavor: default 2026-02-21T01:12:03.187 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-21T01:12:03.187 INFO:teuthology.task.install:extra packages: [] 2026-02-21T01:12:03.187 DEBUG:teuthology.orchestra.run.trial106:> sudo apt-key list | grep Ceph 2026-02-21T01:12:03.258 INFO:teuthology.orchestra.run.trial106.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-21T01:12:03.286 INFO:teuthology.orchestra.run.trial106.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-21T01:12:03.286 INFO:teuthology.orchestra.run.trial106.stdout:uid [ unknown] Ceph.com (release key) 2026-02-21T01:12:03.287 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-21T01:12:03.287 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=36d388f27052dd940bdbaf543bb05bdba0acc155 2026-02-21T01:12:03.422 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default/ 2026-02-21T01:12:03.422 INFO:teuthology.task.install.deb:Package version is 20.2.0-681-g36d388f2-1jammy 2026-02-21T01:12:03.538 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:12:03.538 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-21T01:12:03.547 DEBUG:teuthology.orchestra.run.trial106:> sudo apt-get update 2026-02-21T01:12:03.700 INFO:teuthology.orchestra.run.trial106.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy InRelease 2026-02-21T01:12:03.712 INFO:teuthology.orchestra.run.trial106.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-21T01:12:03.724 INFO:teuthology.orchestra.run.trial106.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-21T01:12:03.733 INFO:teuthology.orchestra.run.trial106.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-21T01:12:03.736 INFO:teuthology.orchestra.run.trial106.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.8 kB] 2026-02-21T01:12:03.737 INFO:teuthology.orchestra.run.trial106.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-21T01:12:03.752 INFO:teuthology.orchestra.run.trial106.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-21T01:12:03.757 INFO:teuthology.orchestra.run.trial106.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-21T01:12:04.075 INFO:teuthology.orchestra.run.trial106.stdout:Fetched 26.5 kB in 0s (72.3 kB/s) 2026-02-21T01:12:04.611 INFO:teuthology.orchestra.run.trial106.stdout:Reading package lists... 2026-02-21T01:12:04.626 DEBUG:teuthology.orchestra.run.trial106:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-681-g36d388f2-1jammy cephadm=20.2.0-681-g36d388f2-1jammy ceph-mds=20.2.0-681-g36d388f2-1jammy ceph-mgr=20.2.0-681-g36d388f2-1jammy ceph-common=20.2.0-681-g36d388f2-1jammy ceph-fuse=20.2.0-681-g36d388f2-1jammy ceph-test=20.2.0-681-g36d388f2-1jammy ceph-volume=20.2.0-681-g36d388f2-1jammy radosgw=20.2.0-681-g36d388f2-1jammy python3-rados=20.2.0-681-g36d388f2-1jammy python3-rgw=20.2.0-681-g36d388f2-1jammy python3-cephfs=20.2.0-681-g36d388f2-1jammy python3-rbd=20.2.0-681-g36d388f2-1jammy libcephfs2=20.2.0-681-g36d388f2-1jammy libcephfs-dev=20.2.0-681-g36d388f2-1jammy librados2=20.2.0-681-g36d388f2-1jammy librbd1=20.2.0-681-g36d388f2-1jammy rbd-fuse=20.2.0-681-g36d388f2-1jammy 2026-02-21T01:12:04.660 INFO:teuthology.orchestra.run.trial106.stdout:Reading package lists... 2026-02-21T01:12:04.770 INFO:teuthology.orchestra.run.trial106.stdout:Building dependency tree... 2026-02-21T01:12:04.771 INFO:teuthology.orchestra.run.trial106.stdout:Reading state information... 2026-02-21T01:12:04.865 INFO:teuthology.orchestra.run.trial106.stdout:The following packages were automatically installed and are no longer required: 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: sg3-utils-udev 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout:The following additional packages will be installed: 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-21T01:12:04.866 INFO:teuthology.orchestra.run.trial106.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-21T01:12:04.867 INFO:teuthology.orchestra.run.trial106.stdout: socat xmlstarlet 2026-02-21T01:12:04.868 INFO:teuthology.orchestra.run.trial106.stdout:Suggested packages: 2026-02-21T01:12:04.868 INFO:teuthology.orchestra.run.trial106.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-21T01:12:04.868 INFO:teuthology.orchestra.run.trial106.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-21T01:12:04.868 INFO:teuthology.orchestra.run.trial106.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-21T01:12:04.868 INFO:teuthology.orchestra.run.trial106.stdout:Recommended packages: 2026-02-21T01:12:04.868 INFO:teuthology.orchestra.run.trial106.stdout: btrfs-tools 2026-02-21T01:12:04.899 INFO:teuthology.orchestra.run.trial106.stdout:The following NEW packages will be installed: 2026-02-21T01:12:04.899 INFO:teuthology.orchestra.run.trial106.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-21T01:12:04.899 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-21T01:12:04.900 INFO:teuthology.orchestra.run.trial106.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-21T01:12:04.901 INFO:teuthology.orchestra.run.trial106.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-21T01:12:04.901 INFO:teuthology.orchestra.run.trial106.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-21T01:12:04.901 INFO:teuthology.orchestra.run.trial106.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-21T01:12:04.901 INFO:teuthology.orchestra.run.trial106.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-21T01:12:04.901 INFO:teuthology.orchestra.run.trial106.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-21T01:12:04.901 INFO:teuthology.orchestra.run.trial106.stdout: socat xmlstarlet 2026-02-21T01:12:04.976 INFO:teuthology.orchestra.run.trial106.stdout:0 upgraded, 86 newly installed, 0 to remove and 13 not upgraded. 2026-02-21T01:12:04.976 INFO:teuthology.orchestra.run.trial106.stdout:Need to get 280 MB of archives. 2026-02-21T01:12:04.976 INFO:teuthology.orchestra.run.trial106.stdout:After this operation, 1119 MB of additional disk space will be used. 2026-02-21T01:12:04.976 INFO:teuthology.orchestra.run.trial106.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-681-g36d388f2-1jammy [3569 kB] 2026-02-21T01:12:04.999 INFO:teuthology.orchestra.run.trial106.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-21T01:12:05.044 INFO:teuthology.orchestra.run.trial106.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-21T01:12:05.044 INFO:teuthology.orchestra.run.trial106.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-21T01:12:05.054 INFO:teuthology.orchestra.run.trial106.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-21T01:12:05.077 INFO:teuthology.orchestra.run.trial106.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-681-g36d388f2-1jammy [2867 kB] 2026-02-21T01:12:05.082 INFO:teuthology.orchestra.run.trial106.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-21T01:12:05.084 INFO:teuthology.orchestra.run.trial106.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-21T01:12:05.089 INFO:teuthology.orchestra.run.trial106.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-21T01:12:05.091 INFO:teuthology.orchestra.run.trial106.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-21T01:12:05.091 INFO:teuthology.orchestra.run.trial106.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-21T01:12:05.092 INFO:teuthology.orchestra.run.trial106.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-21T01:12:05.092 INFO:teuthology.orchestra.run.trial106.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-21T01:12:05.096 INFO:teuthology.orchestra.run.trial106.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-21T01:12:05.097 INFO:teuthology.orchestra.run.trial106.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-21T01:12:05.097 INFO:teuthology.orchestra.run.trial106.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-21T01:12:05.103 INFO:teuthology.orchestra.run.trial106.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-21T01:12:05.104 INFO:teuthology.orchestra.run.trial106.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-21T01:12:05.104 INFO:teuthology.orchestra.run.trial106.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-21T01:12:05.104 INFO:teuthology.orchestra.run.trial106.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-21T01:12:05.105 INFO:teuthology.orchestra.run.trial106.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-21T01:12:05.105 INFO:teuthology.orchestra.run.trial106.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-21T01:12:05.113 INFO:teuthology.orchestra.run.trial106.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-21T01:12:05.115 INFO:teuthology.orchestra.run.trial106.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-21T01:12:05.115 INFO:teuthology.orchestra.run.trial106.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-21T01:12:05.116 INFO:teuthology.orchestra.run.trial106.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-681-g36d388f2-1jammy [833 kB] 2026-02-21T01:12:05.116 INFO:teuthology.orchestra.run.trial106.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-21T01:12:05.123 INFO:teuthology.orchestra.run.trial106.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-21T01:12:05.123 INFO:teuthology.orchestra.run.trial106.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-21T01:12:05.124 INFO:teuthology.orchestra.run.trial106.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-21T01:12:05.129 INFO:teuthology.orchestra.run.trial106.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-681-g36d388f2-1jammy [364 kB] 2026-02-21T01:12:05.134 INFO:teuthology.orchestra.run.trial106.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-681-g36d388f2-1jammy [32.9 kB] 2026-02-21T01:12:05.135 INFO:teuthology.orchestra.run.trial106.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-681-g36d388f2-1jammy [184 kB] 2026-02-21T01:12:05.137 INFO:teuthology.orchestra.run.trial106.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-21T01:12:05.138 INFO:teuthology.orchestra.run.trial106.stdout:Get:35 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-681-g36d388f2-1jammy [83.9 kB] 2026-02-21T01:12:05.139 INFO:teuthology.orchestra.run.trial106.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-21T01:12:05.139 INFO:teuthology.orchestra.run.trial106.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-21T01:12:05.139 INFO:teuthology.orchestra.run.trial106.stdout:Get:38 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-681-g36d388f2-1jammy [341 kB] 2026-02-21T01:12:05.145 INFO:teuthology.orchestra.run.trial106.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-681-g36d388f2-1jammy [8686 kB] 2026-02-21T01:12:05.150 INFO:teuthology.orchestra.run.trial106.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-21T01:12:05.150 INFO:teuthology.orchestra.run.trial106.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-21T01:12:05.150 INFO:teuthology.orchestra.run.trial106.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-21T01:12:05.150 INFO:teuthology.orchestra.run.trial106.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-21T01:12:05.150 INFO:teuthology.orchestra.run.trial106.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-21T01:12:05.151 INFO:teuthology.orchestra.run.trial106.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-21T01:12:05.152 INFO:teuthology.orchestra.run.trial106.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-21T01:12:05.153 INFO:teuthology.orchestra.run.trial106.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-21T01:12:05.154 INFO:teuthology.orchestra.run.trial106.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-21T01:12:05.161 INFO:teuthology.orchestra.run.trial106.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-21T01:12:05.162 INFO:teuthology.orchestra.run.trial106.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-21T01:12:05.164 INFO:teuthology.orchestra.run.trial106.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-21T01:12:05.168 INFO:teuthology.orchestra.run.trial106.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-21T01:12:05.170 INFO:teuthology.orchestra.run.trial106.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-21T01:12:05.171 INFO:teuthology.orchestra.run.trial106.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-21T01:12:05.171 INFO:teuthology.orchestra.run.trial106.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-21T01:12:05.172 INFO:teuthology.orchestra.run.trial106.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-21T01:12:05.173 INFO:teuthology.orchestra.run.trial106.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-21T01:12:05.177 INFO:teuthology.orchestra.run.trial106.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-21T01:12:05.180 INFO:teuthology.orchestra.run.trial106.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-21T01:12:05.181 INFO:teuthology.orchestra.run.trial106.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-21T01:12:05.181 INFO:teuthology.orchestra.run.trial106.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-21T01:12:05.190 INFO:teuthology.orchestra.run.trial106.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-21T01:12:05.288 INFO:teuthology.orchestra.run.trial106.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-681-g36d388f2-1jammy [112 kB] 2026-02-21T01:12:05.289 INFO:teuthology.orchestra.run.trial106.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-681-g36d388f2-1jammy [261 kB] 2026-02-21T01:12:05.294 INFO:teuthology.orchestra.run.trial106.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-681-g36d388f2-1jammy [29.3 MB] 2026-02-21T01:12:05.779 INFO:teuthology.orchestra.run.trial106.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-681-g36d388f2-1jammy [5417 kB] 2026-02-21T01:12:05.864 INFO:teuthology.orchestra.run.trial106.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-681-g36d388f2-1jammy [247 kB] 2026-02-21T01:12:05.867 INFO:teuthology.orchestra.run.trial106.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-681-g36d388f2-1jammy [124 kB] 2026-02-21T01:12:05.869 INFO:teuthology.orchestra.run.trial106.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-681-g36d388f2-1jammy [906 kB] 2026-02-21T01:12:05.883 INFO:teuthology.orchestra.run.trial106.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-681-g36d388f2-1jammy [6405 kB] 2026-02-21T01:12:05.990 INFO:teuthology.orchestra.run.trial106.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-681-g36d388f2-1jammy [21.7 MB] 2026-02-21T01:12:06.347 INFO:teuthology.orchestra.run.trial106.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-681-g36d388f2-1jammy [14.1 kB] 2026-02-21T01:12:06.348 INFO:teuthology.orchestra.run.trial106.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-681-g36d388f2-1jammy [957 kB] 2026-02-21T01:12:06.360 INFO:teuthology.orchestra.run.trial106.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-681-g36d388f2-1jammy [2345 kB] 2026-02-21T01:12:06.399 INFO:teuthology.orchestra.run.trial106.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-681-g36d388f2-1jammy [1049 kB] 2026-02-21T01:12:06.416 INFO:teuthology.orchestra.run.trial106.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-681-g36d388f2-1jammy [179 kB] 2026-02-21T01:12:06.419 INFO:teuthology.orchestra.run.trial106.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-681-g36d388f2-1jammy [45.5 MB] 2026-02-21T01:12:07.171 INFO:teuthology.orchestra.run.trial106.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-681-g36d388f2-1jammy [8625 kB] 2026-02-21T01:12:07.306 INFO:teuthology.orchestra.run.trial106.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-681-g36d388f2-1jammy [14.2 kB] 2026-02-21T01:12:07.307 INFO:teuthology.orchestra.run.trial106.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-681-g36d388f2-1jammy [99.5 MB] 2026-02-21T01:12:08.951 INFO:teuthology.orchestra.run.trial106.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-681-g36d388f2-1jammy [135 kB] 2026-02-21T01:12:08.952 INFO:teuthology.orchestra.run.trial106.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-681-g36d388f2-1jammy [43.3 kB] 2026-02-21T01:12:08.952 INFO:teuthology.orchestra.run.trial106.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-681-g36d388f2-1jammy [30.7 kB] 2026-02-21T01:12:08.953 INFO:teuthology.orchestra.run.trial106.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-681-g36d388f2-1jammy [41.5 kB] 2026-02-21T01:12:08.953 INFO:teuthology.orchestra.run.trial106.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-681-g36d388f2-1jammy [25.1 MB] 2026-02-21T01:12:09.354 INFO:teuthology.orchestra.run.trial106.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-yuri3-testing-2026-02-19-1610-tentacle/36d388f27052dd940bdbaf543bb05bdba0acc155/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-681-g36d388f2-1jammy [97.5 kB] 2026-02-21T01:12:09.632 INFO:teuthology.orchestra.run.trial106.stdout:Fetched 280 MB in 4s (62.9 MB/s) 2026-02-21T01:12:09.648 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-21T01:12:09.803 INFO:teuthology.orchestra.run.trial106.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-21T01:12:09.805 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-21T01:12:09.807 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-21T01:12:09.828 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-21T01:12:09.841 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-21T01:12:09.841 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-21T01:12:09.858 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-21T01:12:09.871 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-21T01:12:09.871 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-21T01:12:09.890 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-21T01:12:09.899 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-21T01:12:09.904 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-21T01:12:09.935 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-21T01:12:09.943 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-21T01:12:09.943 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-21T01:12:09.959 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-21T01:12:09.967 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-21T01:12:09.968 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-21T01:12:09.988 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-21T01:12:09.996 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-21T01:12:09.996 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-21T01:12:10.013 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package librados2. 2026-02-21T01:12:10.021 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../07-librados2_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.022 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking librados2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.073 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libnbd0. 2026-02-21T01:12:10.081 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-21T01:12:10.082 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-21T01:12:10.096 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package librbd1. 2026-02-21T01:12:10.104 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../09-librbd1_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.105 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking librbd1 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.143 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libcephfs2. 2026-02-21T01:12:10.151 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.151 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libcephfs2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.170 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-rados. 2026-02-21T01:12:10.178 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../11-python3-rados_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.178 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-rados (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.195 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-21T01:12:10.203 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:10.204 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-ceph-argparse (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.218 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-cephfs. 2026-02-21T01:12:10.227 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.227 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-cephfs (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.243 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-21T01:12:10.251 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:10.251 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-ceph-common (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.270 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-21T01:12:10.278 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-21T01:12:10.278 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-21T01:12:10.294 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-prettytable. 2026-02-21T01:12:10.302 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-21T01:12:10.302 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-21T01:12:10.317 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-rbd. 2026-02-21T01:12:10.325 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.325 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-rbd (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.342 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-21T01:12:10.350 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-21T01:12:10.351 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-21T01:12:10.370 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package librgw2. 2026-02-21T01:12:10.378 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../19-librgw2_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.378 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking librgw2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.470 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-rgw. 2026-02-21T01:12:10.479 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.479 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-rgw (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.494 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-21T01:12:10.502 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-21T01:12:10.502 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-21T01:12:10.517 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libradosstriper1. 2026-02-21T01:12:10.525 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.526 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libradosstriper1 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.542 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-common. 2026-02-21T01:12:10.550 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../23-ceph-common_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.551 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-common (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.793 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-base. 2026-02-21T01:12:10.802 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../24-ceph-base_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:10.806 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-base (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:10.876 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-21T01:12:10.885 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-21T01:12:10.886 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-21T01:12:10.899 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-cheroot. 2026-02-21T01:12:10.907 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-21T01:12:10.908 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-21T01:12:10.925 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-21T01:12:10.933 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-21T01:12:10.934 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-21T01:12:10.948 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-21T01:12:10.957 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-21T01:12:10.957 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-21T01:12:10.971 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-21T01:12:10.979 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-21T01:12:10.980 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-21T01:12:10.994 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-tempora. 2026-02-21T01:12:11.003 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-21T01:12:11.003 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-21T01:12:11.018 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-portend. 2026-02-21T01:12:11.027 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-21T01:12:11.027 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-21T01:12:11.041 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-21T01:12:11.050 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-21T01:12:11.050 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-21T01:12:11.064 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-21T01:12:11.073 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-21T01:12:11.073 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-21T01:12:11.097 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-natsort. 2026-02-21T01:12:11.105 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-21T01:12:11.106 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-21T01:12:11.121 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-21T01:12:11.130 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:11.130 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mgr-modules-core (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.157 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-21T01:12:11.165 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.166 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.181 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mgr. 2026-02-21T01:12:11.189 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.190 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mgr (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.210 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mon. 2026-02-21T01:12:11.218 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.219 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mon (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.279 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-21T01:12:11.288 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-21T01:12:11.288 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-21T01:12:11.304 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-osd. 2026-02-21T01:12:11.313 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.313 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-osd (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.469 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph. 2026-02-21T01:12:11.477 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../41-ceph_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.477 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.491 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-fuse. 2026-02-21T01:12:11.500 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.501 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-fuse (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.522 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mds. 2026-02-21T01:12:11.530 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.530 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mds (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.562 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package cephadm. 2026-02-21T01:12:11.571 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../44-cephadm_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:11.571 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking cephadm (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.587 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-21T01:12:11.596 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-21T01:12:11.596 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-21T01:12:11.617 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-21T01:12:11.625 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:11.625 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mgr-cephadm (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:11.646 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-21T01:12:11.654 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-21T01:12:11.655 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-21T01:12:11.669 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-routes. 2026-02-21T01:12:11.677 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-21T01:12:11.678 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-21T01:12:11.698 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-21T01:12:11.706 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:11.706 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mgr-dashboard (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:12.125 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-21T01:12:12.134 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-21T01:12:12.135 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-21T01:12:12.173 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-joblib. 2026-02-21T01:12:12.181 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-21T01:12:12.181 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-21T01:12:12.208 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-21T01:12:12.216 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-21T01:12:12.217 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-21T01:12:12.233 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-sklearn. 2026-02-21T01:12:12.242 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-21T01:12:12.242 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-21T01:12:12.325 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-21T01:12:12.334 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:12.334 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:12.510 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-cachetools. 2026-02-21T01:12:12.518 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-21T01:12:12.518 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-21T01:12:12.533 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-rsa. 2026-02-21T01:12:12.542 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-21T01:12:12.542 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-21T01:12:12.559 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-google-auth. 2026-02-21T01:12:12.569 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-21T01:12:12.569 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-21T01:12:12.587 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-21T01:12:12.596 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-21T01:12:12.596 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-21T01:12:12.612 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-websocket. 2026-02-21T01:12:12.620 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-21T01:12:12.621 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-21T01:12:12.638 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-21T01:12:12.647 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-21T01:12:12.647 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-21T01:12:12.742 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-21T01:12:12.751 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:12.751 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:12.765 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-21T01:12:12.774 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-21T01:12:12.775 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-21T01:12:12.791 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-21T01:12:12.799 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-21T01:12:12.800 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-21T01:12:12.815 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package jq. 2026-02-21T01:12:12.824 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-21T01:12:12.824 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-21T01:12:12.839 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package socat. 2026-02-21T01:12:12.847 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-21T01:12:12.848 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-21T01:12:12.867 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package xmlstarlet. 2026-02-21T01:12:12.876 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-21T01:12:12.876 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-21T01:12:12.911 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-test. 2026-02-21T01:12:12.920 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../67-ceph-test_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:12.920 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-test (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:13.766 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package ceph-volume. 2026-02-21T01:12:13.774 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-681-g36d388f2-1jammy_all.deb ... 2026-02-21T01:12:13.775 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking ceph-volume (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:13.799 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-21T01:12:13.808 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:13.809 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libcephfs-daemon (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:13.823 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-21T01:12:13.832 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:13.832 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libcephfs-proxy2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:13.846 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-21T01:12:13.855 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:13.855 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking libcephfs-dev (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:13.871 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package nvme-cli. 2026-02-21T01:12:13.880 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-21T01:12:13.881 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-21T01:12:13.912 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-21T01:12:13.921 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-21T01:12:13.922 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-21T01:12:13.951 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-21T01:12:13.960 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-21T01:12:13.960 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-21T01:12:13.974 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-pluggy. 2026-02-21T01:12:13.983 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-21T01:12:13.984 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-21T01:12:14.001 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-psutil. 2026-02-21T01:12:14.009 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-21T01:12:14.010 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-21T01:12:14.028 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-py. 2026-02-21T01:12:14.037 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-21T01:12:14.037 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-21T01:12:14.058 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-pygments. 2026-02-21T01:12:14.067 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-21T01:12:14.067 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-21T01:12:14.109 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-toml. 2026-02-21T01:12:14.118 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-21T01:12:14.118 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-21T01:12:14.133 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-pytest. 2026-02-21T01:12:14.141 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-21T01:12:14.141 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-21T01:12:14.163 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-simplejson. 2026-02-21T01:12:14.172 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-21T01:12:14.172 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-21T01:12:14.190 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package python3-webob. 2026-02-21T01:12:14.199 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-21T01:12:14.199 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-21T01:12:14.222 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-21T01:12:14.231 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-21T01:12:14.231 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-21T01:12:14.298 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package radosgw. 2026-02-21T01:12:14.307 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../84-radosgw_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:14.308 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking radosgw (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:14.519 INFO:teuthology.orchestra.run.trial106.stdout:Selecting previously unselected package rbd-fuse. 2026-02-21T01:12:14.528 INFO:teuthology.orchestra.run.trial106.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-681-g36d388f2-1jammy_amd64.deb ... 2026-02-21T01:12:14.528 INFO:teuthology.orchestra.run.trial106.stdout:Unpacking rbd-fuse (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:14.556 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-21T01:12:14.620 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-21T01:12:14.621 INFO:teuthology.orchestra.run.trial106.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-21T01:12:14.692 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-21T01:12:14.975 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-21T01:12:15.366 INFO:teuthology.orchestra.run.trial106.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-21T01:12:15.377 INFO:teuthology.orchestra.run.trial106.stdout:Setting up cephadm (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:15.420 INFO:teuthology.orchestra.run.trial106.stdout:Adding system user cephadm....done 2026-02-21T01:12:15.430 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-21T01:12:15.504 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-21T01:12:15.505 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-21T01:12:15.581 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-21T01:12:15.656 INFO:teuthology.orchestra.run.trial106.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-21T01:12:15.656 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-21T01:12:15.751 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-21T01:12:15.875 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-21T01:12:15.951 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-21T01:12:16.016 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-ceph-argparse (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:16.082 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-21T01:12:16.083 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-21T01:12:16.084 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-21T01:12:16.085 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-21T01:12:16.086 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-21T01:12:16.209 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-21T01:12:16.285 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libcephfs-proxy2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:16.286 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-21T01:12:16.366 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-21T01:12:16.457 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-21T01:12:16.702 INFO:teuthology.orchestra.run.trial106.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-21T01:12:16.702 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-21T01:12:16.797 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-21T01:12:16.920 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-21T01:12:17.001 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-21T01:12:17.073 INFO:teuthology.orchestra.run.trial106.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-21T01:12:17.074 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-ceph-common (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:17.161 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-21T01:12:17.657 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-21T01:12:17.661 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-21T01:12:17.736 INFO:teuthology.orchestra.run.trial106.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-21T01:12:17.737 INFO:teuthology.orchestra.run.trial106.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-21T01:12:17.738 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-21T01:12:17.811 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-21T01:12:17.884 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-21T01:12:17.885 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-21T01:12:17.962 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-21T01:12:18.034 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-21T01:12:18.115 INFO:teuthology.orchestra.run.trial106.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-21T01:12:18.116 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-21T01:12:18.199 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-21T01:12:18.200 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-21T01:12:18.276 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-21T01:12:18.359 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-21T01:12:18.432 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-21T01:12:18.433 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-21T01:12:18.568 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-21T01:12:18.641 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-21T01:12:18.641 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-21T01:12:18.722 INFO:teuthology.orchestra.run.trial106.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-21T01:12:18.722 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-21T01:12:18.853 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-21T01:12:18.854 INFO:teuthology.orchestra.run.trial106.stdout:Setting up librados2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:18.855 INFO:teuthology.orchestra.run.trial106.stdout:Setting up librgw2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:18.856 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libsqlite3-mod-ceph (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:18.857 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-21T01:12:19.377 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libcephfs2 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.378 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libradosstriper1 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.379 INFO:teuthology.orchestra.run.trial106.stdout:Setting up librbd1 (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.380 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mgr-modules-core (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.381 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-fuse (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.452 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-21T01:12:19.452 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-21T01:12:19.873 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libcephfs-dev (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.874 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-rados (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.875 INFO:teuthology.orchestra.run.trial106.stdout:Setting up libcephfs-daemon (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.876 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-rbd (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.877 INFO:teuthology.orchestra.run.trial106.stdout:Setting up rbd-fuse (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.878 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-rgw (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.879 INFO:teuthology.orchestra.run.trial106.stdout:Setting up python3-cephfs (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.880 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-common (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:19.906 INFO:teuthology.orchestra.run.trial106.stdout:Adding group ceph....done 2026-02-21T01:12:19.943 INFO:teuthology.orchestra.run.trial106.stdout:Adding system user ceph....done 2026-02-21T01:12:19.952 INFO:teuthology.orchestra.run.trial106.stdout:Setting system user ceph properties....done 2026-02-21T01:12:19.958 INFO:teuthology.orchestra.run.trial106.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-21T01:12:20.046 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-21T01:12:20.314 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-21T01:12:20.734 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-test (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:20.735 INFO:teuthology.orchestra.run.trial106.stdout:Setting up radosgw (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:21.004 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-21T01:12:21.004 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-21T01:12:21.393 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-base (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:21.510 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-21T01:12:21.907 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mds (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:21.967 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-21T01:12:21.968 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-21T01:12:22.369 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mgr (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:22.445 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-21T01:12:22.446 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-21T01:12:22.852 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-osd (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:22.927 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-21T01:12:22.927 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-21T01:12:23.325 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mgr-k8sevents (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.336 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.345 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mon (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.419 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-21T01:12:23.419 INFO:teuthology.orchestra.run.trial106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-21T01:12:23.828 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mgr-cephadm (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.839 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.840 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-mgr-dashboard (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.850 INFO:teuthology.orchestra.run.trial106.stdout:Setting up ceph-volume (20.2.0-681-g36d388f2-1jammy) ... 2026-02-21T01:12:23.960 INFO:teuthology.orchestra.run.trial106.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-21T01:12:24.044 INFO:teuthology.orchestra.run.trial106.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-21T01:12:24.714 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-21T01:12:24.714 INFO:teuthology.orchestra.run.trial106.stdout:Running kernel seems to be up-to-date. 2026-02-21T01:12:24.714 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout:The processor microcode seems to be up-to-date. 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout:No services need to be restarted. 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout:No containers need to be restarted. 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout:No user sessions are running outdated binaries. 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout: 2026-02-21T01:12:24.715 INFO:teuthology.orchestra.run.trial106.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-21T01:12:25.523 INFO:teuthology.orchestra.run.trial106.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-21T01:12:25.530 DEBUG:teuthology.parallel:result is None 2026-02-21T01:12:25.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=36d388f27052dd940bdbaf543bb05bdba0acc155 2026-02-21T01:12:25.667 DEBUG:teuthology.orchestra.run.trial106:> dpkg-query -W -f '${Version}' ceph 2026-02-21T01:12:25.677 INFO:teuthology.orchestra.run.trial106.stdout:20.2.0-681-g36d388f2-1jammy 2026-02-21T01:12:25.677 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-681-g36d388f2-1jammy 2026-02-21T01:12:25.677 INFO:teuthology.task.install:The correct ceph version 20.2.0-681-g36d388f2-1jammy is installed. 2026-02-21T01:12:25.679 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-21T01:12:25.679 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:12:25.679 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-21T01:12:25.727 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-21T01:12:25.727 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:12:25.727 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/usr/bin/daemon-helper 2026-02-21T01:12:25.775 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-21T01:12:25.827 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-21T01:12:25.827 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:12:25.828 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-21T01:12:25.879 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-21T01:12:25.931 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-21T01:12:25.931 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:12:25.931 DEBUG:teuthology.orchestra.run.trial106:> sudo dd of=/usr/bin/stdin-killer 2026-02-21T01:12:25.982 DEBUG:teuthology.orchestra.run.trial106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-21T01:12:26.030 INFO:teuthology.run_tasks:Running task workunit... 2026-02-21T01:12:26.046 INFO:tasks.workunit:Pulling workunits from ref 36d388f27052dd940bdbaf543bb05bdba0acc155 2026-02-21T01:12:26.046 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-21T01:12:26.046 INFO:tasks.workunit:timeout=3h 2026-02-21T01:12:26.046 INFO:tasks.workunit:cleanup=True 2026-02-21T01:12:26.046 DEBUG:teuthology.orchestra.run.trial106:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-21T01:12:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T01:12:26.074 INFO:teuthology.orchestra.run.trial106.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-21T01:12:26.074 DEBUG:teuthology.orchestra.run.trial106:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-21T01:12:26.118 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-21T01:12:26.119 DEBUG:teuthology.orchestra.run.trial106:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-21T01:12:26.163 DEBUG:teuthology.orchestra.run.trial106:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 36d388f27052dd940bdbaf543bb05bdba0acc155 2026-02-21T01:12:26.213 INFO:tasks.workunit.client.0.trial106.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-21T01:12:48.553 INFO:tasks.workunit.client.0.trial106.stderr:Note: switching to '36d388f27052dd940bdbaf543bb05bdba0acc155'. 2026-02-21T01:12:48.553 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.553 INFO:tasks.workunit.client.0.trial106.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-21T01:12:48.553 INFO:tasks.workunit.client.0.trial106.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-21T01:12:48.553 INFO:tasks.workunit.client.0.trial106.stderr:state without impacting any branches by switching back to a branch. 2026-02-21T01:12:48.553 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr: git switch -c 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr:Or undo this operation with: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr: git switch - 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.554 INFO:tasks.workunit.client.0.trial106.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-21T01:12:48.555 INFO:tasks.workunit.client.0.trial106.stderr: 2026-02-21T01:12:48.555 INFO:tasks.workunit.client.0.trial106.stderr:HEAD is now at 36d388f2705 Merge branch 'aclamk-rocksdb-binned-cache-perf-counters-tentacle' of https://github.com/aclamk/ceph into wip-yuri3-testing-2026-02-19-1610-tentacle 2026-02-21T01:12:48.562 DEBUG:teuthology.orchestra.run.trial106:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-21T01:12:48.608 DEBUG:teuthology.orchestra.run.trial106:> set -ex 2026-02-21T01:12:48.608 DEBUG:teuthology.orchestra.run.trial106:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-21T01:12:48.655 INFO:tasks.workunit:Running workunits matching misc on client.0... 2026-02-21T01:12:48.657 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2026-02-21T01:12:48.657 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2026-02-21T01:12:48.708 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/mclock-config 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2026-02-21T01:12:48.713 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2026-02-21T01:12:48.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none ' 2026-02-21T01:12:48.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-21T01:12:48.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2026-02-21T01:12:48.714 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2026-02-21T01:12:48.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T01:12:48.716 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2026-02-21T01:12:48.716 INFO:tasks.workunit.client.0.trial106.stderr:TEST_profile_builtin_to_custom 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:TEST_profile_custom_to_builtin 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:TEST_profile_disallow_builtin_params_modify 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:TEST_profile_disallow_builtin_params_override 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:TEST_recovery_limit_adjustment_mclock' 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:12:48.717 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:12:48.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:12:48.719 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:12:48.720 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:12:48.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:12:48.721 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:12:48.721 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:12:48.722 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:12:48.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:12:48.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:12:48.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:12:48.723 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:12:48.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:12:48.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:12:48.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:12:48.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:12:48.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:12:48.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:12:48.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:12:48.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:12:48.729 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:12:48.729 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.729 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:12:48.730 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:12:48.731 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:12:48.732 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:12:48.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:12:48.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:12:48.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:12:48.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:12:48.733 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:12:48.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:12:48.734 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:12:48.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:12:48.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:12:48.735 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:12:48.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:12:48.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:12:48.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:12:48.737 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:12:48.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:12:48.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:12:48.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:12:48.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:12:48.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:12:48.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:12:48.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:12:48.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:12:48.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:12:48.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:12:48.744 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:12:48.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:12:48.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:12:48.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:12:48.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-21T01:12:48.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T01:12:48.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:12:48.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:12:48.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:12:48.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T01:12:48.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T01:12:48.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:12:48.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:12:48.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:12:48.778 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:12:48.778 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.778 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.779 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:12:48.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:12:48.808 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:12:48.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:12:48.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:12:48.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:12:48.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:12:48.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:12:48.810 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:12:48.810 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:12:48.810 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:12:48.811 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:48.811 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.811 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.811 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:12:48.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:12:48.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get fsid 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:12:48.866 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:48.867 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:48.867 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:48.867 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:12:48.868 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:12:48.868 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get mon_host 2026-02-21T01:12:48.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-21T01:12:48.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T01:12:48.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:12:48.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:12:48.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:12:48.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T01:12:48.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:49.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:12:49.029 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:12:49.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:12:49.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:49.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:12:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:12:49.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T01:12:49.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:12:49.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f038a1c1-0bcb-499e-aeaf-467731a99381 2026-02-21T01:12:49.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f038a1c1-0bcb-499e-aeaf-467731a99381' 2026-02-21T01:12:49.056 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 f038a1c1-0bcb-499e-aeaf-467731a99381 2026-02-21T01:12:49.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:12:49.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRBplp4OCbAxAAGBzf2hYiMpGAgr8w/puvZg== 2026-02-21T01:12:49.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRBplp4OCbAxAAGBzf2hYiMpGAgr8w/puvZg=="}' 2026-02-21T01:12:49.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f038a1c1-0bcb-499e-aeaf-467731a99381 -i td/mclock-config/0/new.json 2026-02-21T01:12:49.172 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:12:49.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T01:12:49.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCRBplp4OCbAxAAGBzf2hYiMpGAgr8w/puvZg== --osd-uuid f038a1c1-0bcb-499e-aeaf-467731a99381 2026-02-21T01:12:49.199 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:49.190+0000 7fded3078a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:49.200 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:49.194+0000 7fded3078a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:49.201 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:49.194+0000 7fded3078a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:49.201 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:49.194+0000 7fded3078a40 -1 bdev(0x558723bb0c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:12:49.202 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:49.194+0000 7fded3078a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T01:12:49.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T01:12:49.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:12:49.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:12:49.879 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:12:49.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:12:49.996 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:12:49.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:12:49.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:12:49.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:12:49.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:12:49.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:12:50.020 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:50.014+0000 7f346c3b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:50.022 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:50.014+0000 7f346c3b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:50.024 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:50.018+0000 7f346c3b9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:12:50.102 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:12:50.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:12:50.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:12:50.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:12:50.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:12:50.413 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:50.406+0000 7f346c3b9a40 -1 Falling back to public interface 2026-02-21T01:12:51.041 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:51.034+0000 7f346c3b9a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:12:51.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:12:51.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:12:51.208 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:12:51.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:12:51.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:12:51.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:12:51.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:12:52.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:12:52.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:12:52.319 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:12:52.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:12:52.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:12:52.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:12:52.526 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3381589025,v1:127.0.0.1:6801/3381589025] [v2:127.0.0.1:6802/3381589025,v1:127.0.0.1:6803/3381589025] exists,up f038a1c1-0bcb-499e-aeaf-467731a99381 2026-02-21T01:12:52.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:12:52.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:12:52.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:12:52.527 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T01:12:52.527 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:52.528 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:52.529 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:52.529 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:12:52.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2026-02-21T01:12:52.591 INFO:tasks.workunit.client.0.trial106.stdout:osd_max_backfills: 1 2026-02-21T01:12:52.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2026-02-21T01:12:52.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-21T01:12:52.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-21T01:12:54.808 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T01:12:54.808 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:54.808 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:54.809 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:54.809 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:54.809 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:54.809 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:54.809 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:54.809 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:54.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:54.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:12:54.871 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2026-02-21T01:12:54.871 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-21T01:12:54.871 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-21T01:12:54.871 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:54.872 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:54.873 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:54.873 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T01:12:54.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-21T01:12:54.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-21T01:12:54.934 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:54.935 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T01:12:54.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-21T01:12:54.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-21T01:12:54.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-21T01:12:55.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-21T01:12:55.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-21T01:12:57.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T01:12:57.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:57.437 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:57.437 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:57.437 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:57.438 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:57.438 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:57.438 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:57.438 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:57.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:57.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:12:57.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T01:12:57.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T01:12:57.501 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-21T01:12:57.501 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:57.501 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:57.501 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:57.502 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T01:12:57.561 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T01:12:57.561 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T01:12:57.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-21T01:12:57.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:12:57.563 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:12:57.563 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:12:57.563 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:12:57.564 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:12:57.564 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:57.564 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:57.564 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:12:57.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:12:57.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T01:12:57.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:12:57.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:12:57.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:12:57.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2026-02-21T01:12:58.143 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already down. 2026-02-21T01:12:58.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2026-02-21T01:12:58.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T01:12:58.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:12:58.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:12:58.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:12:58.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:12:58.154 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:12:58.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:12:58.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:12:58.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T01:12:58.366 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/3381589025,v1:127.0.0.1:6801/3381589025] [v2:127.0.0.1:6802/3381589025,v1:127.0.0.1:6803/3381589025] exists f038a1c1-0bcb-499e-aeaf-467731a99381 2026-02-21T01:12:58.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:12:58.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:12:58.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:12:58.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2026-02-21T01:12:58.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/mclock-config 2026-02-21T01:12:58.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/mclock-config/0 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:12:58.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:12:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:12:58.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:12:58.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:12:58.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:12:58.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2026-02-21T01:12:58.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/mclock-config/0 2026-02-21T01:12:58.373 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:12:58.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:12:58.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2026-02-21T01:12:58.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/mclock-config/0/whoami 2026-02-21T01:12:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:12:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:12:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:12:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:12:58.388 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:58.382+0000 7fbdf77bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:58.390 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:58.382+0000 7fbdf77bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:58.390 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:58.386+0000 7fbdf77bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:12:58.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:12:58.590 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:12:58.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:12:58.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:12:58.798 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:12:58.824 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:58.818+0000 7fbdf77bda40 -1 Falling back to public interface 2026-02-21T01:12:59.414 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:12:59.406+0000 7fbdf77bda40 -1 osd.0 6 log_to_monitors true 2026-02-21T01:12:59.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:12:59.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:12:59.800 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:12:59.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:12:59.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:12:59.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:00.009 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6800/4061478676,v1:127.0.0.1:6801/4061478676] [v2:127.0.0.1:6802/4061478676,v1:127.0.0.1:6803/4061478676] exists,up f038a1c1-0bcb-499e-aeaf-467731a99381 2026-02-21T01:13:00.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:00.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:00.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.010 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.011 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:00.011 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:13:00.011 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:13:00.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T01:13:00.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T01:13:00.074 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-21T01:13:00.074 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:13:00.074 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:13:00.074 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:00.074 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:00.075 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:00.075 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.075 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.075 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:00.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:13:00.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T01:13:00.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T01:13:00.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T01:13:00.135 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-21T01:13:00.135 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:13:00.136 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T01:13:00.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:00.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:00.303 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:00.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:00.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:00.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:00.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:00.306 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:00.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:00.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:00.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:00.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:00.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:00.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:00.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:00.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:00.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:00.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:00.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:00.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:00.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:00.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:00.321 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:00.321 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:00.322 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:00.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:00.323 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:00.325 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:00.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:00.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:00.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:00.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:00.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:00.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:00.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:00.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:00.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:00.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:00.334 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:00.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:00.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:00.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:00.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:00.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:00.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:00.337 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:00.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:00.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:00.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:00.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:00.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:00.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:00.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:00.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:00.344 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:00.344 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.344 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.344 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:00.345 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:13:00.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:13:00.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:13:00.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T01:13:00.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T01:13:00.367 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:13:00.367 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:00.367 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:00.368 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:00.368 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.368 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:00.368 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:13:00.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:13:00.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:13:00.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:00.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:00.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:13:00.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:13:00.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:00.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:00.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:00.401 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:00.402 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.402 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.402 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:00.402 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:00.402 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get fsid 2026-02-21T01:13:00.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:13:00.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:00.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:00.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:13:00.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:00.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get mon_host 2026-02-21T01:13:00.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2026-02-21T01:13:00.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T01:13:00.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:13:00.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:13:00.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:13:00.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T01:13:00.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:13:00.628 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:13:00.628 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:00.629 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:00.629 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:00.629 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.629 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.629 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:00.630 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:00.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:00.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:00.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:00.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:00.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:00.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T01:13:00.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:00.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0b626390-bf6d-446c-a05b-5dd57442f8d0 2026-02-21T01:13:00.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0b626390-bf6d-446c-a05b-5dd57442f8d0' 2026-02-21T01:13:00.651 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 0b626390-bf6d-446c-a05b-5dd57442f8d0 2026-02-21T01:13:00.651 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:00.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCcBplpSgU3JxAA6wmNsi5swxof7vRB78l25A== 2026-02-21T01:13:00.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCcBplpSgU3JxAA6wmNsi5swxof7vRB78l25A=="}' 2026-02-21T01:13:00.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0b626390-bf6d-446c-a05b-5dd57442f8d0 -i td/mclock-config/0/new.json 2026-02-21T01:13:00.770 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:00.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T01:13:00.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCcBplpSgU3JxAA6wmNsi5swxof7vRB78l25A== --osd-uuid 0b626390-bf6d-446c-a05b-5dd57442f8d0 2026-02-21T01:13:00.795 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:00.790+0000 7faa5b8a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:00.797 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:00.790+0000 7faa5b8a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:00.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:00.790+0000 7faa5b8a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:00.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:00.790+0000 7faa5b8a6a40 -1 bdev(0x5557dd3cac00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:00.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:00.790+0000 7faa5b8a6a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T01:13:01.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T01:13:01.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:01.481 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:13:01.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:13:01.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:01.592 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:13:01.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:13:01.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:13:01.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:01.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:01.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:01.608 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:01.602+0000 7f24a2307a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:01.609 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:01.602+0000 7f24a2307a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:01.610 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:01.602+0000 7f24a2307a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:01.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:13:01.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:01.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:13:01.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:01.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:01.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:01.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:01.706 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:01.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:01.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:01.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:02.068 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:02.062+0000 7f24a2307a40 -1 Falling back to public interface 2026-02-21T01:13:02.751 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:02.746+0000 7f24a2307a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:13:02.812 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:02.812 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:02.812 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:02.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:02.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:02.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:02.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:03.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:03.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:03.963 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:03.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:13:03.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:03.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:04.185 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4220647132,v1:127.0.0.1:6801/4220647132] [v2:127.0.0.1:6802/4220647132,v1:127.0.0.1:6803/4220647132] exists,up 0b626390-bf6d-446c-a05b-5dd57442f8d0 2026-02-21T01:13:04.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:04.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:04.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:04.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2026-02-21T01:13:04.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2026-02-21T01:13:04.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2026-02-21T01:13:04.404 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-21T01:13:04.404 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T01:13:04.404 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:04.404 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:04.405 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:04.405 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:04.405 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:04.405 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:04.405 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T01:13:04.405 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:04.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2026-02-21T01:13:04.466 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2026-02-21T01:13:04.466 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2026-02-21T01:13:04.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2026-02-21T01:13:04.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2026-02-21T01:13:04.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-21T01:13:04.536 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:13:04.537 INFO:tasks.workunit.client.0.trial106.stdout: "success": "" 2026-02-21T01:13:04.537 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:13:04.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:04.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T01:13:04.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:04.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2026-02-21T01:13:04.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2026-02-21T01:13:04.607 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2026-02-21T01:13:04.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2026-02-21T01:13:04.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2026-02-21T01:13:04.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T01:13:04.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:04.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2026-02-21T01:13:04.670 INFO:tasks.workunit.client.0.trial106.stdout:client_res = 0.300000 2026-02-21T01:13:04.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2026-02-21T01:13:04.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2026-02-21T01:13:04.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2026-02-21T01:13:04.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2026-02-21T01:13:04.672 INFO:tasks.workunit.client.0.trial106.stdout:client_res_new = .400000 2026-02-21T01:13:04.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2026-02-21T01:13:04.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2026-02-21T01:13:04.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T01:13:05.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2026-02-21T01:13:05.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2026-02-21T01:13:05.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2026-02-21T01:13:05.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-21T01:13:05.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:05.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2026-02-21T01:13:05.111 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T01:13:05.111 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:05.111 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:05.112 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:05.113 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.113 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.113 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:05.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T01:13:05.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:05.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2026-02-21T01:13:05.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2026-02-21T01:13:05.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2026-02-21T01:13:05.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-21T01:13:05.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2026-02-21T01:13:05.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:05.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:05.285 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:05.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:05.286 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:05.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:05.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:05.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:05.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:05.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:05.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:05.291 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:05.291 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:05.296 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:05.296 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.296 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.296 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:05.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:05.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:05.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T01:13:05.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:05.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:05.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:05.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:05.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:05.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:05.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:05.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:05.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:05.301 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:05.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:05.302 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:05.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:05.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:05.303 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:05.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:05.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:05.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:05.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:05.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:05.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:05.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:05.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:05.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:05.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:05.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:05.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:05.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:05.314 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:05.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:05.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:05.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:05.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:05.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:05.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:05.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:05.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:05.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:05.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:05.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:05.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.322 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:05.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:05.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:05.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:05.324 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:05.325 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.325 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:05.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:05.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:05.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:05.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:05.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2026-02-21T01:13:05.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2026-02-21T01:13:05.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:05.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:05.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:05.330 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:05.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:05.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:05.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:05.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:05.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:05.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:05.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:05.334 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:05.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:05.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:05.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:05.338 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:05.338 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.338 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:05.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:05.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:05.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:05.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:05.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:05.342 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T01:13:05.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T01:13:05.365 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:13:05.365 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:05.365 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:05.365 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:05.365 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.365 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.366 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:05.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:13:05.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:13:05.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:13:05.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:05.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:05.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:13:05.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:13:05.397 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:05.398 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:05.398 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:05.398 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:05.398 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.398 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.398 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:05.399 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:05.399 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get fsid 2026-02-21T01:13:05.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:13:05.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:05.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:05.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:13:05.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:13:05.455 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:05.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:05.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:05.456 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:05.456 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.456 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:05.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:05.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get mon_host 2026-02-21T01:13:05.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2026-02-21T01:13:05.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T01:13:05.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:13:05.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:13:05.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:13:05.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T01:13:05.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:13:05.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:13:05.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:05.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:05.625 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:05.625 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.625 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:05.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:05.626 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:05.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:05.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:13:05.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T01:13:05.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:05.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c12d3cd-609d-4555-a3a1-ad047d9defbc 2026-02-21T01:13:05.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1c12d3cd-609d-4555-a3a1-ad047d9defbc' 2026-02-21T01:13:05.650 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 1c12d3cd-609d-4555-a3a1-ad047d9defbc 2026-02-21T01:13:05.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:05.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChBplp7qBQJxAACEUiTlYjCM0IUJt1HSGWuw== 2026-02-21T01:13:05.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChBplp7qBQJxAACEUiTlYjCM0IUJt1HSGWuw=="}' 2026-02-21T01:13:05.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c12d3cd-609d-4555-a3a1-ad047d9defbc -i td/mclock-config/0/new.json 2026-02-21T01:13:05.775 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:05.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T01:13:05.786 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQChBplp7qBQJxAACEUiTlYjCM0IUJt1HSGWuw== --osd-uuid 1c12d3cd-609d-4555-a3a1-ad047d9defbc 2026-02-21T01:13:05.804 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:05.798+0000 7f0566c79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:05.805 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:05.798+0000 7f0566c79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:05.806 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:05.798+0000 7f0566c79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:05.806 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:05.802+0000 7f0566c79a40 -1 bdev(0x55c39d324c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:05.806 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:05.802+0000 7f0566c79a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T01:13:06.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T01:13:06.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:06.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:13:06.552 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:13:06.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:06.676 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:13:06.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:13:06.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:13:06.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:06.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:06.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:06.699 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:06.694+0000 7fd02a08ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:06.701 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:06.694+0000 7fd02a08ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:06.703 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:06.698+0000 7fd02a08ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:06.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:13:06.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:06.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:13:06.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:06.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:06.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:06.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:06.789 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:06.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:06.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:06.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:07.148 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:07.142+0000 7fd02a08ea40 -1 Falling back to public interface 2026-02-21T01:13:07.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:07.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:07.899 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:07.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:07.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:07.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:08.101 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:08.094+0000 7fd02a08ea40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:13:08.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:09.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:09.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:09.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:13:09.114 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:09.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:09.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:09.327 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1350783871,v1:127.0.0.1:6801/1350783871] [v2:127.0.0.1:6802/1350783871,v1:127.0.0.1:6803/1350783871] exists,up 1c12d3cd-609d-4555-a3a1-ad047d9defbc 2026-02-21T01:13:09.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:09.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:09.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:09.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2026-02-21T01:13:09.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2026-02-21T01:13:09.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2026-02-21T01:13:09.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2026-02-21T01:13:09.540 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-21T01:13:09.540 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:09.540 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:09.540 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:09.540 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:09.540 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:09.541 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:09.541 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:09.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:09.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:09.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2026-02-21T01:13:09.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2026-02-21T01:13:09.602 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2026-02-21T01:13:09.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2026-02-21T01:13:09.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-21T01:13:09.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-21T01:13:09.670 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:13:09.670 INFO:tasks.workunit.client.0.trial106.stdout: "success": "" 2026-02-21T01:13:09.670 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:13:09.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:09.679 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:09.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:09.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:09.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2026-02-21T01:13:09.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2026-02-21T01:13:09.739 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2026-02-21T01:13:09.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2026-02-21T01:13:09.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2026-02-21T01:13:09.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2026-02-21T01:13:09.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:09.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2026-02-21T01:13:09.740 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:09.740 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:09.740 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:09.741 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:09.741 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:09.741 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:09.741 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:09.741 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:09.741 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:09.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2026-02-21T01:13:09.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2026-02-21T01:13:09.804 INFO:tasks.workunit.client.0.trial106.stdout:Original client_res for osd.0 = 0.300000 2026-02-21T01:13:09.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2026-02-21T01:13:09.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:09.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2026-02-21T01:13:09.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2026-02-21T01:13:09.805 INFO:tasks.workunit.client.0.trial106.stdout:client_res_new = .400000 2026-02-21T01:13:09.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2026-02-21T01:13:10.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T01:13:10.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2026-02-21T01:13:10.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-21T01:13:10.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:10.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:10.257 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:10.258 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:10.258 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:10.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2026-02-21T01:13:10.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-21T01:13:10.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:10.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T01:13:10.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2026-02-21T01:13:10.398 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:13:10.399 INFO:tasks.workunit.client.0.trial106.stdout: "success": "" 2026-02-21T01:13:10.399 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:13:10.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-21T01:13:10.408 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:10.408 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:10.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:10.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2026-02-21T01:13:10.468 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2026-02-21T01:13:10.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-21T01:13:10.468 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T01:13:10.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2026-02-21T01:13:10.686 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-21T01:13:10.686 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:10.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T01:13:10.688 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:10.688 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2026-02-21T01:13:10.689 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:10.689 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:10.689 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:10.690 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:10.690 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:10.690 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:10.690 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:10.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:10.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:10.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2026-02-21T01:13:10.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2026-02-21T01:13:10.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:10.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T01:13:10.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2026-02-21T01:13:10.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2026-02-21T01:13:15.980 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T01:13:16.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2026-02-21T01:13:16.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2026-02-21T01:13:16.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:16.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.201 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.202 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:16.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T01:13:16.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:16.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2026-02-21T01:13:16.262 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2026-02-21T01:13:16.263 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:16.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:16.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:16.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:16.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:16.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:16.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:16.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:16.376 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:16.376 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.376 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:16.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:16.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:16.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:16.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.383 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:16.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:16.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:16.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:16.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:16.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:16.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:16.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:16.388 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:16.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:16.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:16.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:16.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:16.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:16.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:16.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:16.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:16.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:16.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:16.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:16.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:16.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:16.399 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:16.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:16.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:16.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:16.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:16.402 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:16.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:16.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:16.403 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:16.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:16.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:16.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:16.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.407 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:16.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:16.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:16.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:16.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:16.410 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.410 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:16.411 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:16.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:16.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:16.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:16.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:16.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:16.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:16.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:16.415 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:16.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:16.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:16.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:16.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:16.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:16.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:16.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:16.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:16.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:16.421 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:16.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:16.423 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:16.423 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.423 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:16.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:16.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:16.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:16.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:16.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:16.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:13:16.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:13:16.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T01:13:16.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T01:13:16.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:13:16.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:16.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:16.450 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:16.450 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.450 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:16.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:13:16.478 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:13:16.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:13:16.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:16.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:16.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:13:16.480 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:16.481 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:16.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get fsid 2026-02-21T01:13:16.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:13:16.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:16.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:16.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:13:16.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:13:16.538 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:16.538 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:16.538 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:16.538 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:16.538 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.538 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.539 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:16.539 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:16.539 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get mon_host 2026-02-21T01:13:16.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2026-02-21T01:13:16.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T01:13:16.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:13:16.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:13:16.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:13:16.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T01:13:16.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:13:16.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:13:16.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:16.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:16.701 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:16.701 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.701 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:16.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:16.702 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:16.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:16.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:16.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:13:16.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T01:13:16.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:16.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f301f8d2-cf15-4003-8042-0198f28d98d8 2026-02-21T01:13:16.726 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 f301f8d2-cf15-4003-8042-0198f28d98d8 2026-02-21T01:13:16.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f301f8d2-cf15-4003-8042-0198f28d98d8' 2026-02-21T01:13:16.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:16.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCsBplpwv6lKxAAXXF/ca9jfhumAMC8wsRVpQ== 2026-02-21T01:13:16.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCsBplpwv6lKxAAXXF/ca9jfhumAMC8wsRVpQ=="}' 2026-02-21T01:13:16.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f301f8d2-cf15-4003-8042-0198f28d98d8 -i td/mclock-config/0/new.json 2026-02-21T01:13:16.846 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:16.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T01:13:16.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCsBplpwv6lKxAAXXF/ca9jfhumAMC8wsRVpQ== --osd-uuid f301f8d2-cf15-4003-8042-0198f28d98d8 2026-02-21T01:13:16.872 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:16.866+0000 7fccb9ac6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:16.873 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:16.866+0000 7fccb9ac6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:16.874 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:16.866+0000 7fccb9ac6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:16.874 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:16.866+0000 7fccb9ac6a40 -1 bdev(0x561b994a2c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:16.874 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:16.870+0000 7fccb9ac6a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T01:13:17.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T01:13:17.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:17.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:13:17.611 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:13:17.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:17.729 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:13:17.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:13:17.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:13:17.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:17.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:17.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:17.754 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:17.746+0000 7faaaf3b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:17.756 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:17.750+0000 7faaaf3b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:17.757 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:17.750+0000 7faaaf3b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:17.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:17.942 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:18.200 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:18.194+0000 7faaaf3b5a40 -1 Falling back to public interface 2026-02-21T01:13:18.807 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:18.802+0000 7faaaf3b5a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:13:18.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:18.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:18.943 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:18.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:18.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:18.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:19.056 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:20.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:20.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:20.058 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:20.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:13:20.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:20.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:20.271 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3939071655,v1:127.0.0.1:6801/3939071655] [v2:127.0.0.1:6802/3939071655,v1:127.0.0.1:6803/3939071655] exists,up f301f8d2-cf15-4003-8042-0198f28d98d8 2026-02-21T01:13:20.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:20.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:20.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:20.272 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2026-02-21T01:13:20.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2026-02-21T01:13:20.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:20.482 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:20.483 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:20.483 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-21T01:13:20.483 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2026-02-21T01:13:20.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2026-02-21T01:13:20.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2026-02-21T01:13:20.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2026-02-21T01:13:20.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2026-02-21T01:13:20.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:20.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2026-02-21T01:13:20.544 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-21T01:13:20.544 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:20.544 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:20.545 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:20.545 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:20.545 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:20.545 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:20.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-21T01:13:20.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:20.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2026-02-21T01:13:20.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2026-02-21T01:13:20.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:20.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2026-02-21T01:13:20.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2026-02-21T01:13:20.827 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2026-02-21T01:13:20.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2026-02-21T01:13:20.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2026-02-21T01:13:20.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2026-02-21T01:13:20.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2026-02-21T01:13:22.830 INFO:tasks.workunit.client.0.trial106.stdout:Check configuration values - Attempt#: 0 2026-02-21T01:13:22.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2026-02-21T01:13:22.831 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:23.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2026-02-21T01:13:23.043 INFO:tasks.workunit.client.0.trial106.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2026-02-21T01:13:23.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2026-02-21T01:13:23.044 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2026-02-21T01:13:23.044 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:23.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2026-02-21T01:13:23.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2026-02-21T01:13:23.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:23.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2026-02-21T01:13:23.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2026-02-21T01:13:23.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2026-02-21T01:13:23.254 INFO:tasks.workunit.client.0.trial106.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2026-02-21T01:13:23.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2026-02-21T01:13:23.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2026-02-21T01:13:23.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:23.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2026-02-21T01:13:23.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:23.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-21T01:13:23.258 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:23.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2026-02-21T01:13:23.259 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-21T01:13:23.259 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:23.259 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:23.259 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:23.260 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:23.260 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:23.260 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:23.260 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-21T01:13:23.260 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:23.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2026-02-21T01:13:23.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2026-02-21T01:13:23.320 INFO:tasks.workunit.client.0.trial106.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2026-02-21T01:13:23.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2026-02-21T01:13:23.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:23.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2026-02-21T01:13:23.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:23.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-21T01:13:23.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2026-02-21T01:13:23.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2026-02-21T01:13:23.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:23.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-21T01:13:23.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:23.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2026-02-21T01:13:23.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2026-02-21T01:13:23.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:23.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2026-02-21T01:13:23.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2026-02-21T01:13:23.602 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2026-02-21T01:13:23.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2026-02-21T01:13:23.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2026-02-21T01:13:23.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2026-02-21T01:13:23.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2026-02-21T01:13:25.606 INFO:tasks.workunit.client.0.trial106.stdout:Check configuration values - Attempt#: 0 2026-02-21T01:13:25.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2026-02-21T01:13:25.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T01:13:25.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2026-02-21T01:13:25.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2026-02-21T01:13:25.819 INFO:tasks.workunit.client.0.trial106.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2026-02-21T01:13:25.819 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2026-02-21T01:13:25.820 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:25.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2026-02-21T01:13:25.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2026-02-21T01:13:25.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2026-02-21T01:13:25.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2026-02-21T01:13:25.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2026-02-21T01:13:26.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2026-02-21T01:13:26.029 INFO:tasks.workunit.client.0.trial106.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2026-02-21T01:13:26.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2026-02-21T01:13:26.030 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2026-02-21T01:13:26.030 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:26.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2026-02-21T01:13:26.032 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:26.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-21T01:13:26.034 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:26.034 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2026-02-21T01:13:26.034 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2026-02-21T01:13:26.034 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:26.034 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:26.035 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:26.035 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.035 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.035 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:26.035 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2026-02-21T01:13:26.035 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:26.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2026-02-21T01:13:26.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2026-02-21T01:13:26.094 INFO:tasks.workunit.client.0.trial106.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2026-02-21T01:13:26.095 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2026-02-21T01:13:26.095 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:26.096 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2026-02-21T01:13:26.096 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2026-02-21T01:13:26.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:26.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:26.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:26.206 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:26.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:26.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:26.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:26.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:26.210 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:26.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:26.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:26.211 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:26.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:26.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:26.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:26.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:26.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:26.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:26.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:26.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:26.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:26.224 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:26.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:26.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:26.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:26.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:26.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:26.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:26.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:26.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:26.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:26.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T01:13:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:26.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:26.236 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:26.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:26.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:26.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:26.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:26.239 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:26.240 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.240 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:26.240 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:26.240 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:26.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:26.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:26.244 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:26.244 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:26.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:26.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:26.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:26.247 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:26.247 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.247 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:26.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:26.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:26.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:26.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:26.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2026-02-21T01:13:26.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2026-02-21T01:13:26.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2026-02-21T01:13:26.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:26.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:26.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:26.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:26.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:26.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:26.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:26.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:26.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:26.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:26.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:26.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:26.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:26.257 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:26.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:26.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:26.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:26.261 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:26.261 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.261 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:26.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:26.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:26.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:26.263 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:26.263 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.264 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:26.265 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:26.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:26.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:26.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:26.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2026-02-21T01:13:26.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T01:13:26.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:13:26.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:13:26.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:13:26.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T01:13:26.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:26.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:13:26.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:13:26.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:13:26.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:26.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:26.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:13:26.318 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:26.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:26.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get fsid 2026-02-21T01:13:26.376 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:13:26.376 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:26.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:26.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:13:26.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:13:26.377 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:26.378 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get mon_host 2026-02-21T01:13:26.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2026-02-21T01:13:26.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T01:13:26.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:13:26.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:13:26.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:13:26.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T01:13:26.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:13:26.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:13:26.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:26.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:26.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:26.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:26.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:26.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:26.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:26.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:26.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:13:26.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T01:13:26.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:26.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=423fe273-aca6-43e6-a1a7-f84bf8fe6a0e 2026-02-21T01:13:26.573 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 423fe273-aca6-43e6-a1a7-f84bf8fe6a0e 2026-02-21T01:13:26.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 423fe273-aca6-43e6-a1a7-f84bf8fe6a0e' 2026-02-21T01:13:26.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:26.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2BplpbGSZIhAAu/n9eEXEOMLvyD2ggTjVXQ== 2026-02-21T01:13:26.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2BplpbGSZIhAAu/n9eEXEOMLvyD2ggTjVXQ=="}' 2026-02-21T01:13:26.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 423fe273-aca6-43e6-a1a7-f84bf8fe6a0e -i td/mclock-config/0/new.json 2026-02-21T01:13:26.693 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:26.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T01:13:26.702 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQC2BplpbGSZIhAAu/n9eEXEOMLvyD2ggTjVXQ== --osd-uuid 423fe273-aca6-43e6-a1a7-f84bf8fe6a0e 2026-02-21T01:13:26.717 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:26.710+0000 7efdaf8a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:26.719 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:26.714+0000 7efdaf8a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:26.720 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:26.714+0000 7efdaf8a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:26.720 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:26.714+0000 7efdaf8a1a40 -1 bdev(0x558c3b7ecc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:26.721 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:26.714+0000 7efdaf8a1a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T01:13:27.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T01:13:27.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:27.403 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:13:27.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:13:27.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:27.526 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:13:27.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:13:27.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:13:27.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:27.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:27.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:27.549 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:27.542+0000 7fa6822cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:27.552 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:27.546+0000 7fa6822cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:27.554 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:27.546+0000 7fa6822cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:27.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:13:27.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:27.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:13:27.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:27.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:27.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:27.636 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:27.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:27.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:27.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:27.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:28.080 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:28.073+0000 7fa6822cea40 -1 Falling back to public interface 2026-02-21T01:13:28.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:28.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:28.745 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:28.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:28.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:28.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:28.856 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:29.067 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:29.061+0000 7fa6822cea40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:13:29.857 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:29.857 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:29.858 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:29.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:13:29.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:29.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:30.071 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1438036286,v1:127.0.0.1:6801/1438036286] [v2:127.0.0.1:6802/1438036286,v1:127.0.0.1:6803/1438036286] exists,up 423fe273-aca6-43e6-a1a7-f84bf8fe6a0e 2026-02-21T01:13:30.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:30.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:30.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:30.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2026-02-21T01:13:30.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2026-02-21T01:13:30.283 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2026-02-21T01:13:30.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2026-02-21T01:13:30.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-21T01:13:30.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:30.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:30.285 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:30.285 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:30.285 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:30.285 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:30.285 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-21T01:13:30.285 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T01:13:30.345 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2026-02-21T01:13:30.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2026-02-21T01:13:30.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2026-02-21T01:13:30.348 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:30.348 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2026-02-21T01:13:30.348 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-21T01:13:30.348 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:30.348 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:30.349 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:30.349 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:30.349 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:30.349 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:30.349 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-21T01:13:30.349 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:30.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2026-02-21T01:13:30.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2026-02-21T01:13:30.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:30.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2026-02-21T01:13:30.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2026-02-21T01:13:30.488 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:13:30.488 INFO:tasks.workunit.client.0.trial106.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2026-02-21T01:13:30.488 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:13:30.495 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2026-02-21T01:13:30.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2026-02-21T01:13:30.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2026-02-21T01:13:30.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2026-02-21T01:13:30.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2026-02-21T01:13:32.499 INFO:tasks.workunit.client.0.trial106.stdout:Check configuration values - Attempt#: 0 2026-02-21T01:13:32.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2026-02-21T01:13:32.499 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:32.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2026-02-21T01:13:32.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2026-02-21T01:13:32.701 INFO:tasks.workunit.client.0.trial106.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2026-02-21T01:13:32.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2026-02-21T01:13:32.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:32.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2026-02-21T01:13:32.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2026-02-21T01:13:32.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:32.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2026-02-21T01:13:32.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2026-02-21T01:13:32.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2026-02-21T01:13:32.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2026-02-21T01:13:32.920 INFO:tasks.workunit.client.0.trial106.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2026-02-21T01:13:32.921 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2026-02-21T01:13:32.921 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:32.922 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2026-02-21T01:13:32.922 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:32.923 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-21T01:13:32.924 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:32.924 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2026-02-21T01:13:32.925 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-21T01:13:32.925 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-21T01:13:32.926 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2026-02-21T01:13:32.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2026-02-21T01:13:32.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2026-02-21T01:13:32.986 INFO:tasks.workunit.client.0.trial106.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2026-02-21T01:13:32.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2026-02-21T01:13:32.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:32.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2026-02-21T01:13:32.989 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:32.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-21T01:13:32.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2026-02-21T01:13:32.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2026-02-21T01:13:32.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:32.991 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:32.992 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:32.992 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:32.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-21T01:13:32.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:33.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2026-02-21T01:13:33.051 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2026-02-21T01:13:33.051 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:33.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2026-02-21T01:13:33.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2026-02-21T01:13:33.122 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:13:33.122 INFO:tasks.workunit.client.0.trial106.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2026-02-21T01:13:33.123 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:13:33.130 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2026-02-21T01:13:33.131 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2026-02-21T01:13:33.132 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2026-02-21T01:13:33.132 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2026-02-21T01:13:33.132 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2026-02-21T01:13:35.133 INFO:tasks.workunit.client.0.trial106.stdout:Check configuration values - Attempt#: 0 2026-02-21T01:13:35.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2026-02-21T01:13:35.134 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T01:13:35.345 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2026-02-21T01:13:35.345 INFO:tasks.workunit.client.0.trial106.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2026-02-21T01:13:35.345 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2026-02-21T01:13:35.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2026-02-21T01:13:35.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:35.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2026-02-21T01:13:35.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2026-02-21T01:13:35.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2026-02-21T01:13:35.348 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2026-02-21T01:13:35.348 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2026-02-21T01:13:35.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2026-02-21T01:13:35.558 INFO:tasks.workunit.client.0.trial106.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2026-02-21T01:13:35.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2026-02-21T01:13:35.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2026-02-21T01:13:35.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:35.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2026-02-21T01:13:35.561 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:35.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-21T01:13:35.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2026-02-21T01:13:35.563 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.564 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:35.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2026-02-21T01:13:35.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T01:13:35.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2026-02-21T01:13:35.623 INFO:tasks.workunit.client.0.trial106.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2026-02-21T01:13:35.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2026-02-21T01:13:35.623 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2026-02-21T01:13:35.623 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:35.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2026-02-21T01:13:35.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:35.627 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:35.628 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:35.628 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:35.628 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:35.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:35.736 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:35.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:35.738 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:35.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:35.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:35.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:35.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:35.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:35.741 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:35.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:35.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:35.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:35.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.747 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:35.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:35.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:35.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T01:13:35.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:35.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:35.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:35.749 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:35.749 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:35.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:35.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:35.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:35.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:35.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:35.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:35.752 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:35.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:35.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:35.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:35.755 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.755 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:35.755 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:35.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:35.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:35.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:35.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:35.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:35.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:35.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:35.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:35.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:35.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:35.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:35.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:35.764 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:35.764 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:35.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:35.765 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:35.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:35.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:35.766 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:35.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:35.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:35.768 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:35.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:35.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:35.772 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:35.772 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.772 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:35.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:35.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:35.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:35.775 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:35.775 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.775 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.775 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:35.777 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T01:13:35.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:35.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:35.780 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:35.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:35.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:35.782 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:35.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:35.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:35.783 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:35.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:35.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:35.785 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:35.786 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:35.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:35.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:35.789 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:35.789 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.789 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:35.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:35.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:35.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T01:13:35.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:35.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12027 2026-02-21T01:13:35.793 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:35.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:13:35.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:35.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T01:13:35.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-21T01:13:35.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T01:13:35.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:13:35.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:13:35.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:13:35.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T01:13:35.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:35.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:13:35.845 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:13:35.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:13:35.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:35.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:35.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:13:35.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.847 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:35.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:35.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get fsid 2026-02-21T01:13:35.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:13:35.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:35.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:35.908 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:35.909 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-mon.a.asok 2026-02-21T01:13:35.909 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:35.909 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.12027/ceph-mon.a.asok config get mon_host 2026-02-21T01:13:35.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-21T01:13:35.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T01:13:35.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:13:35.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:13:35.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:13:35.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T01:13:35.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:13:36.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:13:36.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:36.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:36.077 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:36.077 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:36.078 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:36.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:36.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:36.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:36.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T01:13:36.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T01:13:36.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:36.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:36.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:13:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T01:13:36.100 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:36.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=42d1335f-86d3-4b23-bb05-c9db2f9b51e6 2026-02-21T01:13:36.101 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 42d1335f-86d3-4b23-bb05-c9db2f9b51e6 2026-02-21T01:13:36.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 42d1335f-86d3-4b23-bb05-c9db2f9b51e6' 2026-02-21T01:13:36.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:36.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDABplppp+PBhAAms6ZWrviOPfJkR+mpxctqg== 2026-02-21T01:13:36.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDABplppp+PBhAAms6ZWrviOPfJkR+mpxctqg=="}' 2026-02-21T01:13:36.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 42d1335f-86d3-4b23-bb05-c9db2f9b51e6 -i td/mclock-config/0/new.json 2026-02-21T01:13:36.222 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:36.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T01:13:36.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDABplppp+PBhAAms6ZWrviOPfJkR+mpxctqg== --osd-uuid 42d1335f-86d3-4b23-bb05-c9db2f9b51e6 2026-02-21T01:13:36.246 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:36.241+0000 7f1963f50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:36.248 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:36.241+0000 7f1963f50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:36.249 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:36.241+0000 7f1963f50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:36.249 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:36.241+0000 7f1963f50a40 -1 bdev(0x563ae862cc00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:36.249 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:36.241+0000 7f1963f50a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T01:13:36.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T01:13:36.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:36.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:13:36.939 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:13:36.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:37.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:13:37.051 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:13:37.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=ead2aed1-7415-4f5f-9e37-83825d809a8f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.12027/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:13:37.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:37.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:37.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:37.067 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:37.061+0000 7f2dd5d03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:37.068 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:37.061+0000 7f2dd5d03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:37.069 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:37.061+0000 7f2dd5d03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:37.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:37.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:37.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:37.455 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:37.449+0000 7f2dd5d03a40 -1 Falling back to public interface 2026-02-21T01:13:38.099 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:38.093+0000 7f2dd5d03a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:13:38.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:38.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:38.271 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:38.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:38.273 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:38.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:38.421 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:39.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:39.423 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:39.423 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:39.423 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:13:39.423 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:39.423 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:39.637 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4243686843,v1:127.0.0.1:6801/4243686843] [v2:127.0.0.1:6802/4243686843,v1:127.0.0.1:6803/4243686843] exists,up 42d1335f-86d3-4b23-bb05-c9db2f9b51e6 2026-02-21T01:13:39.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:39.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:39.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:39.639 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:13:39.639 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:39.639 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:39.639 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:39.639 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:39.639 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:39.640 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:39.640 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:13:39.640 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-21T01:13:39.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2026-02-21T01:13:39.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2026-02-21T01:13:39.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2026-02-21T01:13:39.699 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-21T01:13:39.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-21T01:13:41.910 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:13:41.911 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:41.911 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:41.912 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:41.912 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:41.912 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:41.912 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:41.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:13:41.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-21T01:13:41.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2026-02-21T01:13:41.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2026-02-21T01:13:41.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-21T01:13:42.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-21T01:13:42.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:44.417 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12027/ceph-osd.0.asok 2026-02-21T01:13:44.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T01:13:44.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.12027/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-21T01:13:44.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2026-02-21T01:13:44.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2026-02-21T01:13:44.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:44.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:44.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:44.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:44.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:44.590 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:44.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:44.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:44.591 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:44.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:44.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:44.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:44.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:44.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:44.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:44.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:44.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:44.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:44.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:44.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T01:13:44.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:44.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:44.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:44.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:44.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:44.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:44.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:44.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:44.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:44.605 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:44.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:44.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:44.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:44.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:44.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:44.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:44.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:44.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:44.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:44.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:44.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:44.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:44.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:44.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:44.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:44.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-21T01:13:44.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/mclock-config 0 2026-02-21T01:13:44.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T01:13:44.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T01:13:44.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T01:13:44.616 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:44.616 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:44.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:44.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:44.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:44.617 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:44.618 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:44.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:44.619 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:44.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:44.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:44.620 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:44.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:44.622 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:44.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:44.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T01:13:44.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T01:13:44.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:44.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12027 2026-02-21T01:13:44.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12027 2026-02-21T01:13:44.626 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:44.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:44.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-21T01:13:44.628 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:13:44.628 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:13:44.683 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2026-02-21T01:13:44.683 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2026-02-21T01:13:44.735 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/network-ping 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2026-02-21T01:13:44.738 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2026-02-21T01:13:44.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none ' 2026-02-21T01:13:44.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2026-02-21T01:13:44.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2026-02-21T01:13:44.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2026-02-21T01:13:44.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2026-02-21T01:13:44.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2026-02-21T01:13:44.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:TEST_network_ping_test2' 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:13:44.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T01:13:44.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:13:44.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:13:44.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:13:44.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:13:44.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:13:44.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:13:44.745 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:13:44.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:13:44.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:13:44.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:13:44.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:13:44.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:13:44.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:13:44.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:13:44.749 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:13:44.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:13:44.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:13:44.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T01:13:44.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:13:44.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:44.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.28998 2026-02-21T01:13:44.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:13:44.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:13:44.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-21T01:13:44.756 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:13:44.756 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.756 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:44.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.28998 2026-02-21T01:13:44.758 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:13:44.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:13:44.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:13:44.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:13:44.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-21T01:13:44.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-21T01:13:44.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-21T01:13:44.786 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:13:44.786 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:44.786 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:44.786 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:44.786 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.786 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:44.787 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:44.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:13:44.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:44.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:44.817 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:44.817 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.817 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:44.817 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mon.a.asok 2026-02-21T01:13:44.817 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:44.817 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.28998/ceph-mon.a.asok config get fsid 2026-02-21T01:13:44.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:13:44.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:13:44.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:13:44.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mon.a.asok 2026-02-21T01:13:44.875 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:13:44.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.28998/ceph-mon.a.asok config get mon_host 2026-02-21T01:13:44.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2026-02-21T01:13:44.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-21T01:13:44.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:13:44.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:13:44.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:13:44.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-21T01:13:44.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:13:45.046 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:13:45.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:45.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:45.047 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:45.047 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:45.047 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:45.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:45.048 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:45.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:45.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:45.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:13:45.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-21T01:13:45.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:45.069 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ec3e95ad-d0e7-4656-a916-21c61cf8e3e3 2026-02-21T01:13:45.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ec3e95ad-d0e7-4656-a916-21c61cf8e3e3' 2026-02-21T01:13:45.070 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 ec3e95ad-d0e7-4656-a916-21c61cf8e3e3 2026-02-21T01:13:45.070 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:45.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDJBplp4MWnBBAA8LmYQLJLgrugMdkdEiRjzw== 2026-02-21T01:13:45.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDJBplp4MWnBBAA8LmYQLJLgrugMdkdEiRjzw=="}' 2026-02-21T01:13:45.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ec3e95ad-d0e7-4656-a916-21c61cf8e3e3 -i td/network-ping/0/new.json 2026-02-21T01:13:45.191 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:45.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-21T01:13:45.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJBplp4MWnBBAA8LmYQLJLgrugMdkdEiRjzw== --osd-uuid ec3e95ad-d0e7-4656-a916-21c61cf8e3e3 2026-02-21T01:13:45.216 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:45.209+0000 7f2a24c96a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:45.218 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:45.213+0000 7f2a24c96a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:45.219 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:45.213+0000 7f2a24c96a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:45.219 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:45.213+0000 7f2a24c96a40 -1 bdev(0x5647b2362c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:45.220 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:45.213+0000 7f2a24c96a40 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-21T01:13:45.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-21T01:13:45.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:45.982 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:13:45.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:13:45.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:46.106 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:13:46.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:13:46.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:13:46.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:46.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:46.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:46.129 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:46.121+0000 7fd1dfb21a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:46.132 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:46.125+0000 7fd1dfb21a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:46.132 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:46.125+0000 7fd1dfb21a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:46.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:46.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:46.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:46.563 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:46.557+0000 7fd1dfb21a40 -1 Falling back to public interface 2026-02-21T01:13:47.262 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:47.257+0000 7fd1dfb21a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:13:47.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:47.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:47.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:47.333 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:47.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:47.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:47.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:48.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:48.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:48.443 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:48.444 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:13:48.444 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:48.444 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:13:48.653 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/775156094,v1:127.0.0.1:6801/775156094] [v2:127.0.0.1:6802/775156094,v1:127.0.0.1:6803/775156094] exists,up ec3e95ad-d0e7-4656-a916-21c61cf8e3e3 2026-02-21T01:13:48.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:48.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:48.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:48.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-21T01:13:48.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:48.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-21T01:13:48.659 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:48.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=116ed09c-43eb-49c3-a2cf-5608b4a0ce87 2026-02-21T01:13:48.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 116ed09c-43eb-49c3-a2cf-5608b4a0ce87' 2026-02-21T01:13:48.660 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 116ed09c-43eb-49c3-a2cf-5608b4a0ce87 2026-02-21T01:13:48.661 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:48.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMBplpTIzNJxAAgCHWPrNGViejv9R1R9z0HA== 2026-02-21T01:13:48.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMBplpTIzNJxAAgCHWPrNGViejv9R1R9z0HA=="}' 2026-02-21T01:13:48.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 116ed09c-43eb-49c3-a2cf-5608b4a0ce87 -i td/network-ping/1/new.json 2026-02-21T01:13:48.889 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:48.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-21T01:13:48.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMBplpTIzNJxAAgCHWPrNGViejv9R1R9z0HA== --osd-uuid 116ed09c-43eb-49c3-a2cf-5608b4a0ce87 2026-02-21T01:13:48.917 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:48.909+0000 7fca72c30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:48.918 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:48.913+0000 7fca72c30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:48.919 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:48.913+0000 7fca72c30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:48.920 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:48.913+0000 7fca72c30a40 -1 bdev(0x55ff65fb8c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:48.920 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:48.913+0000 7fca72c30a40 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-21T01:13:49.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-21T01:13:49.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:49.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:13:49.590 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:13:49.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:49.886 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:13:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:13:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:13:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:49.911 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:49.905+0000 7f6c91f87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:49.913 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:49.905+0000 7f6c91f87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:49.914 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:49.909+0000 7f6c91f87a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:50.092 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:50.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:50.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:13:50.295 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:50.289+0000 7f6c91f87a40 -1 Falling back to public interface 2026-02-21T01:13:50.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:50.909 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:50.901+0000 7f6c91f87a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:13:51.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:51.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:51.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:51.313 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:51.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:51.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:13:51.526 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4289702334,v1:127.0.0.1:6811/4289702334] [v2:127.0.0.1:6812/4289702334,v1:127.0.0.1:6813/4289702334] exists,up 116ed09c-43eb-49c3-a2cf-5608b4a0ce87 2026-02-21T01:13:51.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:51.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:51.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:51.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:13:51.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:51.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:13:51.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:13:51.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-21T01:13:51.533 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:13:51.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c6e7f634-2c47-45d1-9905-fea43d487a5c 2026-02-21T01:13:51.534 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 c6e7f634-2c47-45d1-9905-fea43d487a5c 2026-02-21T01:13:51.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c6e7f634-2c47-45d1-9905-fea43d487a5c' 2026-02-21T01:13:51.534 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:13:51.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPBplp1gZLIBAA0ukilxzfgHZJfruDdjGSHg== 2026-02-21T01:13:51.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPBplp1gZLIBAA0ukilxzfgHZJfruDdjGSHg=="}' 2026-02-21T01:13:51.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c6e7f634-2c47-45d1-9905-fea43d487a5c -i td/network-ping/2/new.json 2026-02-21T01:13:51.768 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:13:51.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-21T01:13:51.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPBplp1gZLIBAA0ukilxzfgHZJfruDdjGSHg== --osd-uuid c6e7f634-2c47-45d1-9905-fea43d487a5c 2026-02-21T01:13:51.795 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:51.789+0000 7fd9495aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:51.797 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:51.789+0000 7fd9495aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:51.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:51.793+0000 7fd9495aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:51.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:51.793+0000 7fd9495aba40 -1 bdev(0x55a14b784c00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:13:51.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:51.793+0000 7fd9495aba40 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-21T01:13:52.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-21T01:13:52.421 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:13:52.422 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:13:52.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:13:52.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:13:52.721 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:13:52.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:13:52.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:13:52.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:13:52.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:13:52.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:13:52.746 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:52.741+0000 7f48d820ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:52.748 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:52.741+0000 7f48d820ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:52.749 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:52.741+0000 7f48d820ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:13:52.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:13:52.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:13:52.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:13:52.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:13:52.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:13:52.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:52.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:13:52.929 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:13:52.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:52.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:13:53.092 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:53.085+0000 7f48d820ca40 -1 Falling back to public interface 2026-02-21T01:13:53.143 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:13:53.709 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:13:53.701+0000 7f48d820ca40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:13:54.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:13:54.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:13:54.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:13:54.144 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:13:54.145 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:13:54.145 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:13:54.350 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2373719524,v1:127.0.0.1:6819/2373719524] [v2:127.0.0.1:6820/2373719524,v1:127.0.0.1:6821/2373719524] exists,up c6e7f634-2c47-45d1-9905-fea43d487a5c 2026-02-21T01:13:54.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:13:54.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:13:54.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:13:54.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2026-02-21T01:13:59.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2026-02-21T01:13:59.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2026-02-21T01:13:59.609 INFO:tasks.workunit.client.0.trial106.stderr:pool 'foo' already exists 2026-02-21T01:13:59.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:14:00.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-21T01:14:00.641 INFO:tasks.workunit.client.0.trial106.stdout:hints = 1 2026-02-21T01:14:00.641 INFO:tasks.workunit.client.0.trial106.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-21T01:14:00.641 INFO:tasks.workunit.client.0.trial106.stdout:Object prefix: benchmark_data_trial106_33730 2026-02-21T01:14:00.641 INFO:tasks.workunit.client.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-21T01:14:00.641 INFO:tasks.workunit.client.0.trial106.stdout: 0 0 0 0 0 0 - 0 2026-02-21T01:14:01.641 INFO:tasks.workunit.client.0.trial106.stdout: 1 16 9993 9977 38.9693 38.9727 0.0013661 0.001601 2026-02-21T01:14:02.641 INFO:tasks.workunit.client.0.trial106.stdout: 2 15 20772 20757 40.5377 42.1094 0.00119094 0.00153964 2026-02-21T01:14:03.641 INFO:tasks.workunit.client.0.trial106.stdout: 3 16 31937 31921 41.5604 43.6094 0.00101347 0.00150199 2026-02-21T01:14:04.641 INFO:tasks.workunit.client.0.trial106.stdout: 4 16 42645 42629 41.6266 41.8281 0.00119928 0.00149976 2026-02-21T01:14:05.641 INFO:tasks.workunit.client.0.trial106.stdout: 5 16 53841 53825 42.0474 43.7344 0.00100017 0.00148474 2026-02-21T01:14:06.642 INFO:tasks.workunit.client.0.trial106.stdout: 6 16 64874 64858 42.2219 43.0977 0.00161779 0.00147873 2026-02-21T01:14:07.641 INFO:tasks.workunit.client.0.trial106.stdout: 7 16 76033 76017 42.4169 43.5898 0.00163528 0.00147199 2026-02-21T01:14:08.641 INFO:tasks.workunit.client.0.trial106.stdout: 8 16 86273 86257 42.1144 40 0.00168517 0.00148256 2026-02-21T01:14:09.641 INFO:tasks.workunit.client.0.trial106.stdout: 9 16 95724 95708 41.5367 36.918 0.00178636 0.00150317 2026-02-21T01:14:10.641 INFO:tasks.workunit.client.0.trial106.stdout: 10 4 105153 105149 41.0707 36.8789 0.00103557 0.00152033 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Total time run: 10.0011 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Total writes made: 105153 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Write size: 4096 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Object size: 4096 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Bandwidth (MB/sec): 41.0708 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Stddev Bandwidth: 2.70825 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Max bandwidth (MB/sec): 43.7344 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Min bandwidth (MB/sec): 36.8789 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Average IOPS: 10514 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Stddev IOPS: 693.312 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Max IOPS: 11196 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Min IOPS: 9441 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Average Latency(s): 0.00152032 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Stddev Latency(s): 0.000689937 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Max latency(s): 0.0101535 2026-02-21T01:14:11.642 INFO:tasks.workunit.client.0.trial106.stdout:Min latency(s): 0.000477984 2026-02-21T01:14:11.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2026-02-21T01:14:21.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2026-02-21T01:14:21.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:14:21.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:14:22.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:14:22.581 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:14:22.581 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:14:22.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:14:22.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:14:22.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:14:22.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-21T01:14:22.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-21T01:14:22.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-21T01:14:22.837 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:14:22.837 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:14:23.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672969 2026-02-21T01:14:23.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672969 2026-02-21T01:14:23.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2026-02-21T01:14:23.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:14:23.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:14:23.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542152 2026-02-21T01:14:23.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542152 2026-02-21T01:14:23.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-60129542152' 2026-02-21T01:14:23.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:14:23.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-21T01:14:23.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:14:23.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:14:23.679 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-21T01:14:23.679 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:14:23.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-21T01:14:23.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-21T01:14:23.681 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836489 2026-02-21T01:14:23.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:14:23.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-21T01:14:23.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:14:23.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672969 2026-02-21T01:14:23.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:14:23.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:14:23.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672969 2026-02-21T01:14:23.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:14:23.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672969 2026-02-21T01:14:23.909 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672969 2026-02-21T01:14:23.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2026-02-21T01:14:23.910 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:14:24.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672969 2026-02-21T01:14:24.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:14:25.125 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:14:25.126 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:14:25.344 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672969 2026-02-21T01:14:25.344 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:14:26.344 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:14:26.345 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:14:26.563 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672969 -lt 42949672969 2026-02-21T01:14:26.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:14:26.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542152 2026-02-21T01:14:26.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:14:26.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:14:26.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542152 2026-02-21T01:14:26.567 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:14:26.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542152 2026-02-21T01:14:26.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2026-02-21T01:14:26.568 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542152 2026-02-21T01:14:26.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:14:26.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542152 2026-02-21T01:14:26.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:26.799 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:14:26.800 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:14:26.800 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:14:26.800 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:26.800 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:26.800 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:26.800 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:14:26.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:26.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 2026-02-21T01:14:26.852 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:26.852 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 1000, 2026-02-21T01:14:26.852 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:14:26.852 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:26.859 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:26.859 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:26.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:26.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:26.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:26.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T01:14:26.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:26.883 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:14:26.883 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:26.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 2026-02-21T01:14:26.930 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:26.931 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 1000, 2026-02-21T01:14:26.931 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:14:26.931 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:26.937 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:26.937 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:26.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:26.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:26.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:26.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T01:14:26.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:26.960 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:14:26.960 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:14:26.960 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:14:26.960 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:26.961 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:26.961 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:26.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:14:26.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:26.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 0 2026-02-21T01:14:27.011 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:27.011 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 0, 2026-02-21T01:14:27.011 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [ 2026-02-21T01:14:27.011 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.011 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:25 2026", 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.178, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.196, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.192 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.278, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.338, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.338 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.278, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.338, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.338 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.157 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:25 2026", 2026-02-21T01:14:27.012 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.176, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.190, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.163 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.245, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.342, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.342 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.245, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.342, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.342 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.164 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:22 2026", 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:27.013 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.153, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.190, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.188 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.175, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.371, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.371 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.175, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.371, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.371 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.133 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:22 2026", 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.152, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.170, 2026-02-21T01:14:27.014 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.188 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.170, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.306, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.306 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.170, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.306, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.306 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.144 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:14:27.015 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:27.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:27.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:27.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2026-02-21T01:14:27.030 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:27.030 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:27.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:27.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:27.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:14:27.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:27.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 0 2026-02-21T01:14:27.089 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:27.089 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 0, 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [ 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:24 2026", 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.173, 2026-02-21T01:14:27.090 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.447, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.297 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.158, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.138, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.138 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.207, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 5.485, 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 5.485 2026-02-21T01:14:27.091 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.172 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:24 2026", 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.143, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.403, 2026-02-21T01:14:27.092 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.267 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.127, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.119, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.119 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.155, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 5.538, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 5.538 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.093 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.145 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:21 2026", 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.157, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.193, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.324 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.094 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.129, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.123, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.123 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.180, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.383, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 1.206 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.165 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.095 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:21 2026", 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.120, 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.141, 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.281 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.096 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.107, 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.097, 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.097 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.138, 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.224, 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 1.160 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.186 2026-02-21T01:14:27.097 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:19 2026", 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.199, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.220, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.262 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.098 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.151, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.140, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.140 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.329, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.444, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.444 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.201 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.099 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:19 2026", 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.162, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.188, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.248 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.121, 2026-02-21T01:14:27.100 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.113, 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.113 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.294, 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.427, 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.427 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.209 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:24 2026", 2026-02-21T01:14:27.101 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.151, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.157, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.209 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.131, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.116, 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.116 2026-02-21T01:14:27.102 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.171, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.211, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.318 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.161 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:24 2026", 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:27.103 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.154, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.161, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.198 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.121, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.104, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.104 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.188, 2026-02-21T01:14:27.104 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.281, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.281 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.188 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:20 2026", 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.133, 2026-02-21T01:14:27.105 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.198, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.192 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.113, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.113, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.113 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.161, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.338, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.338 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.106 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.164 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:22 2026", 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.153, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.190, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.188 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.107 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.122, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.118, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.118 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.175, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.371, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.371 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.136 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.108 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:22 2026", 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.152, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.170, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.188 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.134, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.123, 2026-02-21T01:14:27.109 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.123 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.170, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.306, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.306 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.148 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:20 2026", 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:27.110 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.167, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.185, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.161 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.139, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.133, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.133 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.111 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.184, 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.342, 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.342 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.172 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:14:27.112 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:27.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:27.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:27.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2026-02-21T01:14:27.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:27.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:27.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:27.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2026-02-21T01:14:47.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2026-02-21T01:14:47.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:14:47.119 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:14:47.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:14:47.324 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:14:47.324 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:14:47.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:14:47.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:14:47.325 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:14:47.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-21T01:14:47.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-21T01:14:47.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-21T01:14:47.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:14:47.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:14:47.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672976 2026-02-21T01:14:47.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672976 2026-02-21T01:14:47.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-42949672976' 2026-02-21T01:14:47.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:14:47.486 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:14:47.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542159 2026-02-21T01:14:47.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542159 2026-02-21T01:14:47.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-42949672976 2-60129542159' 2026-02-21T01:14:47.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:14:47.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-21T01:14:47.567 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:14:47.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:14:47.568 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-21T01:14:47.568 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:14:47.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-21T01:14:47.570 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836496 2026-02-21T01:14:47.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-21T01:14:47.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:14:47.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-21T01:14:47.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:14:48.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:14:48.785 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:14:48.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-21T01:14:48.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:14:50.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:14:50.001 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:14:50.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836496 2026-02-21T01:14:50.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:14:50.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672976 2026-02-21T01:14:50.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:14:50.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:14:50.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672976 2026-02-21T01:14:50.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:14:50.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672976 2026-02-21T01:14:50.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-21T01:14:50.222 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672976 2026-02-21T01:14:50.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:14:50.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-21T01:14:50.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:14:50.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542159 2026-02-21T01:14:50.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:14:50.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:14:50.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542159 2026-02-21T01:14:50.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:14:50.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542159 2026-02-21T01:14:50.433 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542159 2026-02-21T01:14:50.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2026-02-21T01:14:50.433 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:14:50.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542159 -lt 60129542159 2026-02-21T01:14:50.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:50.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:14:50.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:14:50.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:14:50.656 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:50.656 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:50.657 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:50.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:14:50.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:50.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 2026-02-21T01:14:50.711 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:50.711 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 1000, 2026-02-21T01:14:50.711 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:14:50.712 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:50.718 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.719 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:50.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:50.729 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.730 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:50.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T01:14:50.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:50.742 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:50.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 2026-02-21T01:14:50.792 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:50.792 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 1000, 2026-02-21T01:14:50.792 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:14:50.793 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:50.800 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.800 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:50.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:50.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:50.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T01:14:50.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:50.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:14:50.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:14:50.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:14:50.822 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:50.822 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:50.822 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:50.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:14:50.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:50.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 0 2026-02-21T01:14:50.876 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 0, 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [ 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:50 2026", 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:50.877 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.170, 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.147, 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.177 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.195, 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.207, 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.338 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.878 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.195, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.207, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.338 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.187 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:50 2026", 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:50.879 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.172, 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.150, 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.163 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.203, 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.214, 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.342 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.880 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.203, 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.214, 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.342 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.195 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:47 2026", 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.881 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.128, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.137, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.171 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.138, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.251, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.371 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.882 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.138, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.251, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.371 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.152 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:47 2026", 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.883 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.116, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.130, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.166 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.126, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.218, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.306 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.126, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.218, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.306 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.884 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.171 2026-02-21T01:14:50.885 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:14:50.885 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:14:50.885 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:50.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:50.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2026-02-21T01:14:50.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:50.904 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:50.904 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:14:50.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:14:50.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:14:50.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:14:50.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:14:50.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:14:50.906 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:14:50.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:14:50.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:14:50.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 0 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 0, 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [ 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:44 2026", 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.954 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.151, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.167, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.282 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.129, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.129, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.129 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.160, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.217, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 5.485 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.144 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:44 2026", 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.108, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.134, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.249 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.091, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.091, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.091 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.116, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.243, 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 5.538 2026-02-21T01:14:50.955 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.113 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:46 2026", 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.129, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.142, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.241 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.106, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.106, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.106 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.145, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.226, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 1.206 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.106 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:44 2026", 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.153, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.165, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.223 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.147, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.133, 2026-02-21T01:14:50.956 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.133 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.172, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.220, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.444 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.131 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:44 2026", 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.127, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.140, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.202 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.117, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.116, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.113 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.158, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.209, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.427 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.121 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:46 2026", 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.100, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.112, 2026-02-21T01:14:50.957 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.202 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.093, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.083, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.083 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.110, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.203, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 1.160 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.095 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:44 2026", 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.140, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.156, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.188 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.120, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.116, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.116 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.169, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.225, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.318 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.121 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:44 2026", 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:14:50.958 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.145, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.153, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.181 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.131, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.121, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.104 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.167, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.197, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.281 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.132 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:45 2026", 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.130, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.148, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.179 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.119, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.108, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.108 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.141, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.278, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.338 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.171 2026-02-21T01:14:50.959 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:47 2026", 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.128, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.137, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.171 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.122, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.108, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.108 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.138, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.251, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.371 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.137 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:47 2026", 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.116, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.130, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.166 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.112, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.108, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.108 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.960 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.126, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.218, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.306 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.154 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:14:45 2026", 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.136, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.151, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.162 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.128, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.125, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.125 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.155, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.245, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.342 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.180 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:14:50.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2026-02-21T01:14:50.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:14:50.974 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2026-02-21T01:14:50.985 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:14:50.985 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2026-02-21T01:15:40.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2026-02-21T01:15:40.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:15:40.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:15:41.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:15:41.203 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:15:41.203 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:15:41.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:15:41.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:15:41.204 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:15:41.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-21T01:15:41.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-21T01:15:41.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-21T01:15:41.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:15:41.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:15:41.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672988 2026-02-21T01:15:41.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672988 2026-02-21T01:15:41.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988' 2026-02-21T01:15:41.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:15:41.363 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:15:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542172 2026-02-21T01:15:41.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542172 2026-02-21T01:15:41.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-60129542172' 2026-02-21T01:15:41.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:15:41.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-21T01:15:41.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:15:41.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:15:41.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-21T01:15:41.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:15:41.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-21T01:15:41.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-21T01:15:41.450 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836509 2026-02-21T01:15:41.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:15:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836509 -lt 21474836509 2026-02-21T01:15:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:15:41.664 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672988 2026-02-21T01:15:41.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:15:41.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:15:41.666 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672988 2026-02-21T01:15:41.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:15:41.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672988 2026-02-21T01:15:41.668 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672988 2026-02-21T01:15:41.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2026-02-21T01:15:41.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:15:41.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672988 2026-02-21T01:15:41.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:15:42.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:15:42.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:15:43.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672988 2026-02-21T01:15:43.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:15:44.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:15:44.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:15:44.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672989 -lt 42949672988 2026-02-21T01:15:44.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:15:44.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542172 2026-02-21T01:15:44.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:15:44.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:15:44.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542172 2026-02-21T01:15:44.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:15:44.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542172 2026-02-21T01:15:44.311 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542172 2026-02-21T01:15:44.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542172' 2026-02-21T01:15:44.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:15:44.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542172 -lt 60129542172 2026-02-21T01:15:44.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:44.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:15:44.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:15:44.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 2026-02-21T01:15:44.590 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:15:44.591 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 1000, 2026-02-21T01:15:44.591 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:15:44.591 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:15:44.598 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.598 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:15:44.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:15:44.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2026-02-21T01:15:44.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T01:15:44.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:15:44.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:15:44.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:15:44.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:15:44.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:44.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:44.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:44.623 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:15:44.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:15:44.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 2026-02-21T01:15:44.669 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:15:44.670 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 1000, 2026-02-21T01:15:44.670 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:15:44.670 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:15:44.676 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.676 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:15:44.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:15:44.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2026-02-21T01:15:44.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T01:15:44.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:15:44.698 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:15:44.699 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:15:44.700 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 0 2026-02-21T01:15:44.750 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:15:44.750 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 0, 2026-02-21T01:15:44.750 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [ 2026-02-21T01:15:44.750 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:40 2026", 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.162, 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.148, 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.144 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.751 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.173, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.173, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.208 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.173, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.173, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.208 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.164 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:42 2026", 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.147, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.127, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.127 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.180, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.180, 2026-02-21T01:15:44.752 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.196 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.180, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.180, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.196 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.138 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:40 2026", 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.130, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.138, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.146 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.135, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.170, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.220 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.135, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.170, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.220 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.172 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:42 2026", 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.753 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.145, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.127, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.132 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.181, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.181, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.186 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.181, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.181, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.186 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.180 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:15:44.754 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:15:44.756 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.756 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:15:44.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2026-02-21T01:15:44.767 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.767 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2026-02-21T01:15:44.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:15:44.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:15:44.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:15:44.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:15:44.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:15:44.778 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:44.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:44.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:44.779 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:15:44.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:15:44.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 0 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 0, 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [ 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:39 2026", 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.160, 2026-02-21T01:15:44.825 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.163, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.170 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.137, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.133, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.128 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.188, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.197, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.241 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.170 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:39 2026", 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.164, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.150, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.156 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.148, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.127, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.127 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.183, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.183, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.233 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.826 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.193 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:40 2026", 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.162, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.148, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.144 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.150, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.130, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.108 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.173, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.173, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.208 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.165 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:39 2026", 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.161, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.154, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.161 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.139, 2026-02-21T01:15:44.827 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.127, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.116 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.173, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.192, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.229 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.153 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:40 2026", 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.130, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.138, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.146 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.124, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.121, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.121 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.135, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.170, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.220 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.130 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:39 2026", 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:15:44.828 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.128, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.136, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.145 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.111, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.111, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.111 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.149, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.176, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.201 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.133 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:39 2026", 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 2, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.141, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.135, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.136 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.130, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.116, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.104 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.164, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.179, 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.226 2026-02-21T01:15:44.829 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.185 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:41 2026", 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.138, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.128, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.132 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.117, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.107, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.106 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.167, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.167, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.180 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.167 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:37 2026", 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.131, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.122, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.133 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.105, 2026-02-21T01:15:44.830 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.100, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.100 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.148, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.158, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.251 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.144 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:39 2026", 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 2, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.117, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.119, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.129 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.106, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.100, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.091 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.125, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.147, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.243 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.115 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:37 2026", 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 0, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 1, 2026-02-21T01:15:44.831 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "front", 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.118, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.123, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.129 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.109, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.108, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.099 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.135, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.142, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.218 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.117 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "last update": "Sat Feb 21 01:15:41 2026", 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "stale": false, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "from osd": 1, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "to osd": 0, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "interface": "back", 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "average": { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.122, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.110, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.108 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "min": { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.102, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.094, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.083 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "max": { 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "1min": 0.149, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "5min": 0.149, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "15min": 0.198 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: "last": 0.149 2026-02-21T01:15:44.832 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:15:44.833 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:15:44.833 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:15:44.833 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.833 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T01:15:44.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2026-02-21T01:15:44.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2026-02-21T01:15:44.853 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2026-02-21T01:15:44.853 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:44.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mgr.x.asok 2026-02-21T01:15:44.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:15:44.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-mgr.x.asok dump_osd_network 99 2026-02-21T01:15:44.901 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:15:44.901 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 99, 2026-02-21T01:15:44.901 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:15:44.901 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:15:44.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2026-02-21T01:15:44.917 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2026-02-21T01:15:44.917 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:44.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-osd.0.asok 2026-02-21T01:15:44.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T01:15:44.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.28998/ceph-osd.0.asok dump_osd_network 98 2026-02-21T01:15:44.975 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:15:44.975 INFO:tasks.workunit.client.0.trial106.stdout: "threshold": 98, 2026-02-21T01:15:44.975 INFO:tasks.workunit.client.0.trial106.stdout: "entries": [] 2026-02-21T01:15:44.975 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:15:44.982 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T01:15:44.982 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2026-02-21T01:15:44.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2026-02-21T01:15:44.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2026-02-21T01:15:44.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-21T01:15:44.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T01:15:44.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:15:44.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T01:15:44.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:15:44.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:15:44.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:15:44.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:15:44.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:15:45.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:15:45.306 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:15:45.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:15:45.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:15:45.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:15:45.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:15:45.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:15:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:15:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:15:45.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:15:45.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:15:45.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:15:45.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:15:45.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T01:15:45.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:15:45.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:45.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:45.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.28998 2026-02-21T01:15:45.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:15:45.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:15:45.882 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:15:45.882 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:15:45.883 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:15:45.883 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:15:45.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:15:45.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:15:45.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:15:45.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:15:45.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:15:45.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:15:45.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:15:45.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:15:45.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:15:45.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T01:15:45.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:15:45.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:45.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:45.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.28998 2026-02-21T01:15:45.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:15:45.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:15:45.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-21T01:15:45.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:15:45.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:45.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:45.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.28998 2026-02-21T01:15:45.895 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-21T01:15:45.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-21T01:15:45.917 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:15:45.917 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:15:45.917 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:15:45.918 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:15:45.918 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:45.918 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:45.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:45.918 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:15:45.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:15:45.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:15:45.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:15:45.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:15:45.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:15:45.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:45.950 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mon.a.asok 2026-02-21T01:15:45.951 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:15:45.951 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.28998/ceph-mon.a.asok config get fsid 2026-02-21T01:15:46.006 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:15:46.007 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:15:46.008 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:15:46.008 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:46.008 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:46.008 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.28998/ceph-mon.a.asok 2026-02-21T01:15:46.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:15:46.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.28998/ceph-mon.a.asok config get mon_host 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-21T01:15:46.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:15:46.170 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:15:46.170 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:15:46.170 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:15:46.171 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:15:46.171 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:46.171 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:46.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:46.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:15:46.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:15:46.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:46.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:15:46.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-21T01:15:46.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:15:46.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b9e11a1b-73f5-47c7-8057-6bdf27a569c5 2026-02-21T01:15:46.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b9e11a1b-73f5-47c7-8057-6bdf27a569c5' 2026-02-21T01:15:46.195 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 b9e11a1b-73f5-47c7-8057-6bdf27a569c5 2026-02-21T01:15:46.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:15:46.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCB5lpjERKDBAAWM9z7MtfSE3Q9YysBgCCbA== 2026-02-21T01:15:46.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCB5lpjERKDBAAWM9z7MtfSE3Q9YysBgCCbA=="}' 2026-02-21T01:15:46.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b9e11a1b-73f5-47c7-8057-6bdf27a569c5 -i td/network-ping/0/new.json 2026-02-21T01:15:46.321 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:15:46.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-21T01:15:46.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCB5lpjERKDBAAWM9z7MtfSE3Q9YysBgCCbA== --osd-uuid b9e11a1b-73f5-47c7-8057-6bdf27a569c5 2026-02-21T01:15:46.348 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:46.343+0000 7fa8a2462a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:46.349 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:46.347+0000 7fa8a2462a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:46.350 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:46.347+0000 7fa8a2462a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:46.350 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:46.347+0000 7fa8a2462a40 -1 bdev(0x56108e4a2c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:15:46.350 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:46.347+0000 7fa8a2462a40 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-21T01:15:47.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-21T01:15:47.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:15:47.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:15:47.123 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:15:47.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:15:47.246 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:15:47.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:15:47.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:15:47.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:15:47.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:15:47.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:15:47.271 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:47.267+0000 7fa3228f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:47.273 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:47.271+0000 7fa3228f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:47.275 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:47.271+0000 7fa3228f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:47.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:15:47.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:15:47.626 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:47.623+0000 7fa3228f6a40 -1 Falling back to public interface 2026-02-21T01:15:48.426 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:48.422+0000 7fa3228f6a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:15:48.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:15:48.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:48.463 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:15:48.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:15:48.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:48.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:15:48.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:15:49.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:15:49.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:49.574 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:15:49.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:15:49.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:49.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:15:49.787 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1355194604,v1:127.0.0.1:6801/1355194604] [v2:127.0.0.1:6802/1355194604,v1:127.0.0.1:6803/1355194604] exists,up b9e11a1b-73f5-47c7-8057-6bdf27a569c5 2026-02-21T01:15:49.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:15:49.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:15:49.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:15:49.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-21T01:15:49.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:15:49.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:15:49.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-21T01:15:49.794 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:15:49.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=205fdaf4-b1ac-4f28-abaf-1f6b74b9a626 2026-02-21T01:15:49.794 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 205fdaf4-b1ac-4f28-abaf-1f6b74b9a626 2026-02-21T01:15:49.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 205fdaf4-b1ac-4f28-abaf-1f6b74b9a626' 2026-02-21T01:15:49.795 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:15:49.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFB5lpW1XoLxAAHo1aCDeV4OaLWoK7Fv0V0Q== 2026-02-21T01:15:49.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFB5lpW1XoLxAAHo1aCDeV4OaLWoK7Fv0V0Q=="}' 2026-02-21T01:15:49.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 205fdaf4-b1ac-4f28-abaf-1f6b74b9a626 -i td/network-ping/1/new.json 2026-02-21T01:15:50.032 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:15:50.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-21T01:15:50.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFB5lpW1XoLxAAHo1aCDeV4OaLWoK7Fv0V0Q== --osd-uuid 205fdaf4-b1ac-4f28-abaf-1f6b74b9a626 2026-02-21T01:15:50.067 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:50.062+0000 7f8b3a428a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:50.069 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:50.062+0000 7f8b3a428a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:50.070 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:50.066+0000 7f8b3a428a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:50.070 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:50.066+0000 7f8b3a428a40 -1 bdev(0x55d45586cc00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:15:50.070 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:50.066+0000 7f8b3a428a40 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-21T01:15:50.712 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-21T01:15:50.712 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:15:50.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:15:50.713 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:15:50.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:15:51.005 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:15:51.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:15:51.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:15:51.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:15:51.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:15:51.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:15:51.031 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:51.026+0000 7f5cdd58ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:51.033 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:51.030+0000 7f5cdd58ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:51.034 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:51.030+0000 7f5cdd58ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:51.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:15:51.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:15:51.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:15:51.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:15:51.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:15:51.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:51.216 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:15:51.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:15:51.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:51.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:15:51.407 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:51.402+0000 7f5cdd58ea40 -1 Falling back to public interface 2026-02-21T01:15:51.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:15:52.018 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:52.014+0000 7f5cdd58ea40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:15:52.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:15:52.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:52.431 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:15:52.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:15:52.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:52.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:15:52.647 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/932816457,v1:127.0.0.1:6811/932816457] [v2:127.0.0.1:6812/932816457,v1:127.0.0.1:6813/932816457] exists,up 205fdaf4-b1ac-4f28-abaf-1f6b74b9a626 2026-02-21T01:15:52.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:15:52.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:15:52.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:15:52.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:15:52.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:15:52.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-21T01:15:52.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:15:52.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f1564017-adad-41a0-8bc5-74468c99f0ed 2026-02-21T01:15:52.651 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 f1564017-adad-41a0-8bc5-74468c99f0ed 2026-02-21T01:15:52.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 f1564017-adad-41a0-8bc5-74468c99f0ed' 2026-02-21T01:15:52.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:15:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBIB5lpJ2OFJxAAfagmSm66kDLodPRx77iIPw== 2026-02-21T01:15:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBIB5lpJ2OFJxAAfagmSm66kDLodPRx77iIPw=="}' 2026-02-21T01:15:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f1564017-adad-41a0-8bc5-74468c99f0ed -i td/network-ping/2/new.json 2026-02-21T01:15:52.887 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:15:52.897 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-21T01:15:52.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIB5lpJ2OFJxAAfagmSm66kDLodPRx77iIPw== --osd-uuid f1564017-adad-41a0-8bc5-74468c99f0ed 2026-02-21T01:15:52.916 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:52.910+0000 7fe4b9884a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:52.917 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:52.914+0000 7fe4b9884a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:52.918 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:52.914+0000 7fe4b9884a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:52.919 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:52.914+0000 7fe4b9884a40 -1 bdev(0x55e04ba96c00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:15:52.919 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:52.914+0000 7fe4b9884a40 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-21T01:15:53.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-21T01:15:53.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:15:53.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:15:53.574 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:15:53.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:15:53.866 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:15:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:15:53.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=264b401e-3ff2-412d-8fff-e2c8ad90c1b0 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.28998/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:15:53.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:15:53.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:15:53.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:15:53.891 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:53.886+0000 7f647b6dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:53.893 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:53.890+0000 7f647b6dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:53.894 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:53.890+0000 7f647b6dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:54.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:15:54.278 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:54.274+0000 7f647b6dfa40 -1 Falling back to public interface 2026-02-21T01:15:54.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:15:54.871 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:15:54.866+0000 7f647b6dfa40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:15:55.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:15:55.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:15:55.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:15:55.294 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:15:55.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:15:55.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:15:55.514 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/355733349,v1:127.0.0.1:6819/355733349] [v2:127.0.0.1:6820/355733349,v1:127.0.0.1:6821/355733349] exists,up f1564017-adad-41a0-8bc5-74468c99f0ed 2026-02-21T01:15:55.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:15:55.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:15:55.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:15:55.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2026-02-21T01:16:00.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2026-02-21T01:16:00.777 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'dc1' already exists 2026-02-21T01:16:00.786 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2026-02-21T01:16:01.076 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'dc2' already exists 2026-02-21T01:16:01.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2026-02-21T01:16:01.369 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'dc3' already exists 2026-02-21T01:16:01.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2026-02-21T01:16:01.642 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'rack1' already exists 2026-02-21T01:16:01.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2026-02-21T01:16:01.924 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'rack2' already exists 2026-02-21T01:16:01.934 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2026-02-21T01:16:02.206 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'rack3' already exists 2026-02-21T01:16:02.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2026-02-21T01:16:02.485 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'host1' already exists 2026-02-21T01:16:02.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2026-02-21T01:16:02.763 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'host2' already exists 2026-02-21T01:16:02.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2026-02-21T01:16:03.041 INFO:tasks.workunit.client.0.trial106.stderr:bucket 'host3' already exists 2026-02-21T01:16:03.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2026-02-21T01:16:03.304 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2026-02-21T01:16:03.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2026-02-21T01:16:03.578 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2026-02-21T01:16:03.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2026-02-21T01:16:03.865 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2026-02-21T01:16:03.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2026-02-21T01:16:04.148 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2026-02-21T01:16:04.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2026-02-21T01:16:04.425 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2026-02-21T01:16:04.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2026-02-21T01:16:04.732 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2026-02-21T01:16:04.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2026-02-21T01:16:05.016 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2026-02-21T01:16:05.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2026-02-21T01:16:05.289 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2026-02-21T01:16:05.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2026-02-21T01:16:05.561 INFO:tasks.workunit.client.0.trial106.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2026-02-21T01:16:05.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2026-02-21T01:16:05.837 INFO:tasks.workunit.client.0.trial106.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2026-02-21T01:16:05.848 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2026-02-21T01:16:06.101 INFO:tasks.workunit.client.0.trial106.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2026-02-21T01:16:06.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2026-02-21T01:16:06.366 INFO:tasks.workunit.client.0.trial106.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2026-02-21T01:16:06.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2026-02-21T01:16:06.671 INFO:tasks.workunit.client.0.trial106.stderr:rule myrule already exists 2026-02-21T01:16:06.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2026-02-21T01:16:06.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 16 replicated myrule 2026-02-21T01:16:06.941 INFO:tasks.workunit.client.0.trial106.stderr:pool 'foo' already exists 2026-02-21T01:16:06.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:16:07.953 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-21T01:16:07.974 INFO:tasks.workunit.client.0.trial106.stdout:hints = 1 2026-02-21T01:16:07.974 INFO:tasks.workunit.client.0.trial106.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-21T01:16:07.974 INFO:tasks.workunit.client.0.trial106.stdout:Object prefix: benchmark_data_trial106_39525 2026-02-21T01:16:07.974 INFO:tasks.workunit.client.0.trial106.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-21T01:16:07.974 INFO:tasks.workunit.client.0.trial106.stdout: 0 0 0 0 0 0 - 0 2026-02-21T01:16:08.974 INFO:tasks.workunit.client.0.trial106.stdout: 1 15 10439 10424 40.7144 40.7188 0.00109181 0.00153243 2026-02-21T01:16:09.974 INFO:tasks.workunit.client.0.trial106.stdout: 2 16 20846 20830 40.6796 40.6484 0.00131325 0.00153443 2026-02-21T01:16:10.974 INFO:tasks.workunit.client.0.trial106.stdout: 3 16 31976 31960 41.6107 43.4766 0.00110414 0.00150016 2026-02-21T01:16:11.975 INFO:tasks.workunit.client.0.trial106.stdout: 4 16 43286 43270 42.2521 44.1797 0.000855945 0.00147759 2026-02-21T01:16:12.974 INFO:tasks.workunit.client.0.trial106.stdout: 5 16 54393 54377 42.4784 43.3867 0.00142059 0.00146969 2026-02-21T01:16:13.975 INFO:tasks.workunit.client.0.trial106.stdout: 6 16 65555 65539 42.665 43.6016 0.00112988 0.00146322 2026-02-21T01:16:14.975 INFO:tasks.workunit.client.0.trial106.stdout: 7 15 76630 76615 42.7504 43.2656 0.00127239 0.00146037 2026-02-21T01:16:15.975 INFO:tasks.workunit.client.0.trial106.stdout: 8 16 86936 86920 42.4379 40.2539 0.00163584 0.00147114 2026-02-21T01:16:16.975 INFO:tasks.workunit.client.0.trial106.stdout: 9 16 96332 96316 41.8004 36.7031 0.00136928 0.00149366 2026-02-21T01:16:17.975 INFO:tasks.workunit.client.0.trial106.stdout: 10 1 105765 105764 41.3107 36.9062 0.0015388 0.00151144 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Total time run: 10.001 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Total writes made: 105765 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Write size: 4096 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Object size: 4096 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Bandwidth (MB/sec): 41.3105 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Stddev Bandwidth: 2.76559 2026-02-21T01:16:18.975 INFO:tasks.workunit.client.0.trial106.stdout:Max bandwidth (MB/sec): 44.1797 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Min bandwidth (MB/sec): 36.7031 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Average IOPS: 10575 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Stddev IOPS: 707.991 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Max IOPS: 11310 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Min IOPS: 9396 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Average Latency(s): 0.00151144 2026-02-21T01:16:18.976 INFO:tasks.workunit.client.0.trial106.stdout:Stddev Latency(s): 0.000720442 2026-02-21T01:16:18.977 INFO:tasks.workunit.client.0.trial106.stdout:Max latency(s): 0.00980041 2026-02-21T01:16:18.977 INFO:tasks.workunit.client.0.trial106.stdout:Min latency(s): 0.00051527 2026-02-21T01:16:18.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2026-02-21T01:16:28.983 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2026-02-21T01:16:28.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:16:28.984 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:16:29.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:16:29.199 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:16:29.199 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:16:29.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:16:29.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:16:29.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:16:29.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-21T01:16:29.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-21T01:16:29.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-21T01:16:29.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:16:29.277 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:16:29.357 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-21T01:16:29.357 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-21T01:16:29.357 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672970' 2026-02-21T01:16:29.357 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:16:29.357 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:16:29.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-21T01:16:29.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-21T01:16:29.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-42949672970 2-64424509449' 2026-02-21T01:16:29.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:16:29.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-21T01:16:29.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:16:29.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:16:29.438 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-21T01:16:29.438 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:16:29.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-21T01:16:29.439 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836490 2026-02-21T01:16:29.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-21T01:16:29.440 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:16:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-21T01:16:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:16:30.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:16:30.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:16:30.870 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-21T01:16:30.870 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:16:30.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-21T01:16:30.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:16:30.872 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:16:30.872 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-21T01:16:30.873 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:16:30.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-21T01:16:30.874 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672970 2026-02-21T01:16:30.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-21T01:16:30.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:16:31.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-21T01:16:31.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:16:31.087 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-21T01:16:31.087 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:16:31.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:16:31.088 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-21T01:16:31.089 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:16:31.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-21T01:16:31.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-21T01:16:31.090 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 64424509449 2026-02-21T01:16:31.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:16:31.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509449 -lt 64424509449 2026-02-21T01:16:31.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2026-02-21T01:16:31.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2026-02-21T01:16:31.553 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.389ms); Slow OSD heartbeats on front (longest 0.367ms) 2026-02-21T01:16:31.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2026-02-21T01:16:31.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2026-02-21T01:16:31.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.389ms); Slow OSD heartbeats on front (longest 0.367ms) 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.389ms) 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.389 msec 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.355 msec 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.257 msec 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.233 msec possibly improving 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.187 msec possibly improving 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.183 msec possibly improving 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.367ms) 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.367 msec 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.352 msec 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.303 msec 2026-02-21T01:16:31.850 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.302 msec 2026-02-21T01:16:31.851 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.255 msec possibly improving 2026-02-21T01:16:31.851 INFO:tasks.workunit.client.0.trial106.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.236 msec 2026-02-21T01:16:31.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2026-02-21T01:16:31.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2026-02-21T01:16:31.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2026-02-21T01:16:31.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2026-02-21T01:16:31.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-21T01:16:31.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T01:16:31.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:16:31.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T01:16:31.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:31.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:31.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:31.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:31.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:32.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:32.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:16:32.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:16:32.180 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:16:32.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:16:32.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:16:32.182 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:16:32.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:32.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:16:32.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:16:32.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:32.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:16:32.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:16:32.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T01:16:32.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:16:32.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:32.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:16:32.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.28998 2026-02-21T01:16:32.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:16:32.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:16:32.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-21T01:16:32.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/network-ping 0 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:32.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:32.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:32.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:16:32.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:16:32.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:16:32.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:16:32.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:16:32.892 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:16:32.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:32.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:16:32.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:16:32.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:32.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:16:32.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T01:16:32.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T01:16:32.897 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:16:32.897 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:32.897 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.28998 2026-02-21T01:16:32.897 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.28998 2026-02-21T01:16:32.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:16:32.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:16:32.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-21T01:16:32.899 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:16:32.900 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:16:32.949 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2026-02-21T01:16:32.949 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2026-02-21T01:16:33.002 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:16:33.007 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:16:33.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:16:33.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/ok-to-stop 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2026-02-21T01:16:33.008 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none ' 2026-02-21T01:16:33.009 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none ' 2026-02-21T01:16:33.010 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none ' 2026-02-21T01:16:33.010 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2026-02-21T01:16:33.010 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T01:16:33.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2026-02-21T01:16:33.011 INFO:tasks.workunit.client.0.trial106.stderr:TEST_0_osd 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:TEST_1_mon_checks 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:TEST_2_mons_checks 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:TEST_3_mons_checks 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:TEST_4_mons_checks 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:TEST_5_mons_checks' 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:16:33.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:16:33.013 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:33.013 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:33.013 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:33.013 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:33.013 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:33.013 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:33.014 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:16:33.014 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:16:33.014 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:16:33.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:16:33.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:16:33.015 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:16:33.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:33.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:16:33.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:16:33.017 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:33.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:16:33.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:16:33.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:16:33.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:16:33.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:16:33.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:16:33.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:16:33.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:16:33.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:16:33.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.024 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:16:33.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:16:33.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:16:33.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:16:33.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:16:33.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:16:33.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:16:33.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:16:33.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:16:33.050 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:16:33.050 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:33.050 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:33.050 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:33.050 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.050 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.051 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:33.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:16:33.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:16:33.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:16:33.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:16:33.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:16:33.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:16:33.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:16:33.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:16:33.137 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:16:33.138 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:16:33.139 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:16:33.139 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.140 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.140 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:16:33.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:16:33.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:16:33.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2026-02-21T01:16:33.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-21T01:16:33.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:16:33.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:16:33.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:16:33.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-21T01:16:33.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:33.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:16:33.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:16:33.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:33.331 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:16:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:16:33.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:16:33.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-21T01:16:33.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:16:33.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7322bf0-64b7-41f1-af4b-9e5948add93c 2026-02-21T01:16:33.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e7322bf0-64b7-41f1-af4b-9e5948add93c' 2026-02-21T01:16:33.334 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 e7322bf0-64b7-41f1-af4b-9e5948add93c 2026-02-21T01:16:33.334 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:16:33.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBxB5lp3QFFFBAAhgeUb2tgVJHhfWV+OnlYDw== 2026-02-21T01:16:33.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBxB5lp3QFFFBAAhgeUb2tgVJHhfWV+OnlYDw=="}' 2026-02-21T01:16:33.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7322bf0-64b7-41f1-af4b-9e5948add93c -i td/ok-to-stop/0/new.json 2026-02-21T01:16:33.455 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:16:33.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-21T01:16:33.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxB5lp3QFFFBAAhgeUb2tgVJHhfWV+OnlYDw== --osd-uuid e7322bf0-64b7-41f1-af4b-9e5948add93c 2026-02-21T01:16:33.479 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:33.473+0000 7f504c670a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:33.481 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:33.477+0000 7f504c670a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:33.481 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:33.477+0000 7f504c670a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:33.482 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:33.477+0000 7f504c670a40 -1 bdev(0x56149cf1ec00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:16:33.482 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:33.477+0000 7f504c670a40 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-21T01:16:34.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-21T01:16:34.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:16:34.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:16:34.253 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:16:34.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:16:34.372 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:16:34.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:16:34.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:16:34.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:16:34.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:16:34.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:16:34.397 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:34.393+0000 7f9b69847a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:34.400 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:34.393+0000 7f9b69847a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:34.401 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:34.397+0000 7f9b69847a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:34.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:16:34.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:16:34.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:16:34.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:16:34.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:16:34.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:34.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:16:34.481 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:16:34.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:34.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:16:34.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:16:34.938 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:34.933+0000 7f9b69847a40 -1 Falling back to public interface 2026-02-21T01:16:35.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:16:35.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:35.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:16:35.589 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:16:35.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:35.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:16:35.696 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:16:35.879 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:35.873+0000 7f9b69847a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:16:36.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:16:36.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:36.697 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:16:36.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:16:36.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:36.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4146940759,v1:127.0.0.1:6801/4146940759] [v2:127.0.0.1:6802/4146940759,v1:127.0.0.1:6803/4146940759] exists,up e7322bf0-64b7-41f1-af4b-9e5948add93c 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=a 2026-02-21T01:16:36.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/a 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:36.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:16:36.930 INFO:tasks.workunit.client.0.trial106.stdout:starting mds.a at 2026-02-21T01:16:36.934 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2026-02-21T01:16:37.188 INFO:tasks.workunit.client.0.trial106.stderr:pool 'meta' already exists 2026-02-21T01:16:37.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2026-02-21T01:16:37.497 INFO:tasks.workunit.client.0.trial106.stderr:pool 'data' already exists 2026-02-21T01:16:37.507 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2026-02-21T01:16:37.855 INFO:tasks.workunit.client.0.trial106.stderr:filesystem 'myfs' already exists 2026-02-21T01:16:37.868 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2026-02-21T01:16:42.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2026-02-21T01:16:43.047 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 2026-02-21T01:16:43.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2026-02-21T01:16:43.231 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 2026-02-21T01:16:43.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2026-02-21T01:16:43.503 INFO:tasks.workunit.client.0.trial106.stderr:should be safe to stop [dne] 2026-02-21T01:16:43.512 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2026-02-21T01:16:43.512 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-21T01:16:43.512 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T01:16:43.512 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=b 2026-02-21T01:16:43.512 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T01:16:43.512 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/b 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:43.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:43.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:16:43.530 INFO:tasks.workunit.client.0.trial106.stdout:starting mds.b at 2026-02-21T01:16:43.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2026-02-21T01:16:48.535 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2026-02-21T01:16:48.797 INFO:tasks.workunit.client.0.trial106.stderr:should be safe to stop [a] 2026-02-21T01:16:48.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2026-02-21T01:16:49.085 INFO:tasks.workunit.client.0.trial106.stderr:should be safe to stop [b] 2026-02-21T01:16:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2026-02-21T01:16:49.293 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 4137 2026-02-21T01:16:49.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2026-02-21T01:16:49.568 INFO:tasks.workunit.client.0.trial106.stderr:should be safe to stop [a,dne1,dne2] 2026-02-21T01:16:49.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2026-02-21T01:16:49.867 INFO:tasks.workunit.client.0.trial106.stderr:should be safe to stop [b,dne] 2026-02-21T01:16:49.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2026-02-21T01:16:50.063 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 4137 2026-02-21T01:16:50.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2026-02-21T01:16:50.343 INFO:tasks.workunit.client.0.trial106.stderr:should be safe to stop [dne1,dne2] 2026-02-21T01:16:50.353 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2026-02-21T01:16:50.355 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:50.355 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:50.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:50.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:50.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:50.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:50.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:50.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:50.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:50.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:50.578 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:16:50.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:16:50.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:16:50.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:16:50.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:16:50.581 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:16:50.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:50.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:16:50.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:16:50.583 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:50.584 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:16:50.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:16:50.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:16:50.589 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:16:50.589 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.589 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:16:50.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:16:50.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:16:50.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:16:50.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:16:50.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:16:50.594 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:16:50.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:16:50.595 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:16:50.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:16:50.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:16:50.596 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:16:50.597 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:50.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:16:50.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:16:50.598 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:16:50.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:16:50.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:16:50.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:16:50.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:16:50.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:16:50.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:16:50.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:16:50.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:16:50.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:16:50.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:16:50.605 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:16:50.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:16:50.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:16:50.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:16:50.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:16:50.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:16:50.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:16:50.627 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:16:50.628 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:50.628 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:50.628 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:50.628 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.628 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.628 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:50.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:16:50.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:16:50.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:16:50.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:16:50.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:16:50.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:16:50.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:16:50.659 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:16:50.659 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:16:50.659 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:16:50.660 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:16:50.660 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.660 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.660 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:16:50.660 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:16:50.660 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:16:50.713 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:16:50.714 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:16:50.714 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.714 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.714 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:16:50.714 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:16:50.714 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:16:50.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2026-02-21T01:16:50.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-21T01:16:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:16:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:16:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:16:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-21T01:16:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:50.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:16:50.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:16:50.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2026-02-21T01:16:50.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-21T01:16:50.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:50.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:16:50.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:16:50.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-21T01:16:50.903 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:16:50.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0316eae5-ee62-4930-83fc-6f563f50fda3 2026-02-21T01:16:50.904 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 0316eae5-ee62-4930-83fc-6f563f50fda3 2026-02-21T01:16:50.904 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0316eae5-ee62-4930-83fc-6f563f50fda3' 2026-02-21T01:16:50.904 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:16:50.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCB5lpP9A6NhAAUocMCvOriy7HvepOfNA0Ig== 2026-02-21T01:16:50.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCB5lpP9A6NhAAUocMCvOriy7HvepOfNA0Ig=="}' 2026-02-21T01:16:50.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0316eae5-ee62-4930-83fc-6f563f50fda3 -i td/ok-to-stop/0/new.json 2026-02-21T01:16:51.023 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:16:51.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-21T01:16:51.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCB5lpP9A6NhAAUocMCvOriy7HvepOfNA0Ig== --osd-uuid 0316eae5-ee62-4930-83fc-6f563f50fda3 2026-02-21T01:16:51.048 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.041+0000 7fca3dce5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:51.049 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.045+0000 7fca3dce5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:51.050 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.045+0000 7fca3dce5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:51.050 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.045+0000 7fca3dce5a40 -1 bdev(0x560627336c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:16:51.050 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.045+0000 7fca3dce5a40 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-21T01:16:51.708 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-21T01:16:51.708 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:16:51.709 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:16:51.709 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:16:51.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:16:51.824 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:16:51.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:16:51.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:16:51.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:16:51.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:16:51.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:16:51.839 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.833+0000 7fdf7f461a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:51.840 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.833+0000 7fdf7f461a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:51.842 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:51.837+0000 7fdf7f461a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:51.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:16:51.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:16:51.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:16:51.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:16:51.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:16:51.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:51.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:16:51.941 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:16:51.941 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:51.941 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:16:52.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:16:52.226 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:52.221+0000 7fdf7f461a40 -1 Falling back to public interface 2026-02-21T01:16:53.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:16:53.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:53.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:16:53.051 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:16:53.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:53.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:16:53.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:16:53.306 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:53.301+0000 7fdf7f461a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:16:54.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:16:54.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:54.163 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:16:54.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:16:54.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:54.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:16:54.379 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/646728005,v1:127.0.0.1:6801/646728005] [v2:127.0.0.1:6802/646728005,v1:127.0.0.1:6803/646728005] exists,up 0316eae5-ee62-4930-83fc-6f563f50fda3 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/1 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:16:54.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:54.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/1 2026-02-21T01:16:54.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:16:54.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=721c3097-d21b-4466-a73c-d07fbbf6fd29 2026-02-21T01:16:54.383 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 721c3097-d21b-4466-a73c-d07fbbf6fd29 2026-02-21T01:16:54.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 721c3097-d21b-4466-a73c-d07fbbf6fd29' 2026-02-21T01:16:54.383 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:16:54.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGB5lpXDJ6FxAAixKk48rnkL8nP9kcEeiEoQ== 2026-02-21T01:16:54.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGB5lpXDJ6FxAAixKk48rnkL8nP9kcEeiEoQ=="}' 2026-02-21T01:16:54.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 721c3097-d21b-4466-a73c-d07fbbf6fd29 -i td/ok-to-stop/1/new.json 2026-02-21T01:16:54.615 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:16:54.626 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/1/new.json 2026-02-21T01:16:54.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGB5lpXDJ6FxAAixKk48rnkL8nP9kcEeiEoQ== --osd-uuid 721c3097-d21b-4466-a73c-d07fbbf6fd29 2026-02-21T01:16:54.651 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:54.645+0000 7f0f4f446a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:54.653 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:54.645+0000 7f0f4f446a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:54.654 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:54.649+0000 7f0f4f446a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:54.654 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:54.649+0000 7f0f4f446a40 -1 bdev(0x559ca923ec00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:16:54.654 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:54.649+0000 7f0f4f446a40 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2026-02-21T01:16:55.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/1/keyring 2026-02-21T01:16:55.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:16:55.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:16:55.262 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:16:55.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:16:55.533 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:16:55.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:16:55.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:16:55.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:16:55.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:16:55.535 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:16:55.547 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:55.545+0000 7ffa9cb09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:55.549 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:55.549+0000 7ffa9cb09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:55.550 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:55.549+0000 7ffa9cb09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:55.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:16:55.914 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:55.913+0000 7ffa9cb09a40 -1 Falling back to public interface 2026-02-21T01:16:55.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:16:56.558 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:56.556+0000 7ffa9cb09a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:16:56.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:16:56.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:56.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:16:56.963 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:16:56.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:56.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:16:57.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:16:58.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:16:58.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:58.178 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:16:58.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:16:58.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:58.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:16:58.386 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2763034445,v1:127.0.0.1:6811/2763034445] [v2:127.0.0.1:6812/2763034445,v1:127.0.0.1:6813/2763034445] exists,up 721c3097-d21b-4466-a73c-d07fbbf6fd29 2026-02-21T01:16:58.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/2 2026-02-21T01:16:58.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:16:58.391 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:16:58.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:16:58.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:16:58.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:16:58.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:16:58.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/2 2026-02-21T01:16:58.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:16:58.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c758c19-f1c9-455b-87cd-bb3aabbb7a18 2026-02-21T01:16:58.394 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 6c758c19-f1c9-455b-87cd-bb3aabbb7a18 2026-02-21T01:16:58.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6c758c19-f1c9-455b-87cd-bb3aabbb7a18' 2026-02-21T01:16:58.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:16:58.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCKB5lpvaruFxAAJAvX6F8yVnM4y3HjpnrAgQ== 2026-02-21T01:16:58.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCKB5lpvaruFxAAJAvX6F8yVnM4y3HjpnrAgQ=="}' 2026-02-21T01:16:58.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c758c19-f1c9-455b-87cd-bb3aabbb7a18 -i td/ok-to-stop/2/new.json 2026-02-21T01:16:58.624 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:16:58.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/2/new.json 2026-02-21T01:16:58.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKB5lpvaruFxAAJAvX6F8yVnM4y3HjpnrAgQ== --osd-uuid 6c758c19-f1c9-455b-87cd-bb3aabbb7a18 2026-02-21T01:16:58.659 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:58.656+0000 7fc7adc7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:58.660 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:58.656+0000 7fc7adc7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:58.661 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:58.660+0000 7fc7adc7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:58.662 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:58.660+0000 7fc7adc7fa40 -1 bdev(0x55d857cbac00 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:16:58.662 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:58.660+0000 7fc7adc7fa40 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2026-02-21T01:16:59.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/2/keyring 2026-02-21T01:16:59.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:16:59.297 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:16:59.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:16:59.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:16:59.589 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:16:59.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:16:59.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:16:59.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:16:59.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:16:59.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:16:59.612 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:59.608+0000 7f908d109a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:59.613 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:59.612+0000 7f908d109a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:59.614 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:59.612+0000 7f908d109a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:16:59.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:16:59.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:16:59.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:16:59.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:16:59.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:16:59.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:16:59.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:16:59.806 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:16:59.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:16:59.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:16:59.983 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:16:59.980+0000 7f908d109a40 -1 Falling back to public interface 2026-02-21T01:17:00.014 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:17:00.811 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:00.808+0000 7f908d109a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:17:01.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:17:01.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:17:01.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:17:01.016 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:17:01.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:17:01.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:17:01.231 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:17:02.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:17:02.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:17:02.234 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:17:02.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:17:02.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:17:02.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:17:02.450 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1546254425,v1:127.0.0.1:6819/1546254425] [v2:127.0.0.1:6820/1546254425,v1:127.0.0.1:6821/1546254425] exists,up 6c758c19-f1c9-455b-87cd-bb3aabbb7a18 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:17:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/3 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T01:17:02.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:17:02.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:02.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:02.454 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:02.454 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:17:02.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:17:02.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:17:02.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:17:02.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/3 2026-02-21T01:17:02.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:17:02.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d556139b-7231-4159-b2fa-f20d4918ecbf 2026-02-21T01:17:02.457 INFO:tasks.workunit.client.0.trial106.stdout:add osd3 d556139b-7231-4159-b2fa-f20d4918ecbf 2026-02-21T01:17:02.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 d556139b-7231-4159-b2fa-f20d4918ecbf' 2026-02-21T01:17:02.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:17:02.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOB5lpQ+nBGxAAxZOCLOUM2Uy7HCFuDh2QZw== 2026-02-21T01:17:02.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOB5lpQ+nBGxAAxZOCLOUM2Uy7HCFuDh2QZw=="}' 2026-02-21T01:17:02.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d556139b-7231-4159-b2fa-f20d4918ecbf -i td/ok-to-stop/3/new.json 2026-02-21T01:17:02.681 INFO:tasks.workunit.client.0.trial106.stdout:3 2026-02-21T01:17:02.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/3/new.json 2026-02-21T01:17:02.691 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOB5lpQ+nBGxAAxZOCLOUM2Uy7HCFuDh2QZw== --osd-uuid d556139b-7231-4159-b2fa-f20d4918ecbf 2026-02-21T01:17:02.706 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:02.704+0000 7f6388778a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:17:02.708 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:02.704+0000 7f6388778a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:17:02.709 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:02.708+0000 7f6388778a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:17:02.709 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:02.708+0000 7f6388778a40 -1 bdev(0x55fcf5502c00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2026-02-21T01:17:02.710 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:02.708+0000 7f6388778a40 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2026-02-21T01:17:03.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/3/keyring 2026-02-21T01:17:03.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:17:03.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-21T01:17:03.336 INFO:tasks.workunit.client.0.trial106.stdout:adding osd3 key to auth repository 2026-02-21T01:17:03.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:17:03.624 INFO:tasks.workunit.client.0.trial106.stdout:start osd.3 2026-02-21T01:17:03.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-21T01:17:03.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:17:03.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:17:03.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:17:03.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:17:03.638 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:03.636+0000 7ffa425d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:17:03.639 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:03.636+0000 7ffa425d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:17:03.640 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:03.636+0000 7ffa425d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:17:03.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:17:03.843 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:17:03.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:17:03.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T01:17:04.015 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:04.012+0000 7ffa425d0a40 -1 Falling back to public interface 2026-02-21T01:17:04.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:17:04.651 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:17:04.648+0000 7ffa425d0a40 -1 osd.3 0 log_to_monitors true 2026-02-21T01:17:05.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:17:05.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:17:05.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:17:05.049 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:17:05.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:17:05.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T01:17:05.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:17:06.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:17:06.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:17:06.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:17:06.258 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:17:06.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:17:06.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T01:17:06.473 INFO:tasks.workunit.client.0.trial106.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2232927833,v1:127.0.0.1:6827/2232927833] [v2:127.0.0.1:6828/2232927833,v1:127.0.0.1:6829/2232927833] exists,up d556139b-7231-4159-b2fa-f20d4918ecbf 2026-02-21T01:17:06.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:17:06.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:17:06.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:17:06.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2026-02-21T01:17:06.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2026-02-21T01:17:07.086 INFO:tasks.workunit.client.0.trial106.stderr:pool 'ec' already exists 2026-02-21T01:17:07.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2026-02-21T01:17:07.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:17:07.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:17:07.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:17:07.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:17:07.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:17:07.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:17:07.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:17:07.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:17:07.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:17:07.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:17:07.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:17:07.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:17:07.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:17:07.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:17:07.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr:2 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr:3' 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:07.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:17:07.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-21T01:17:07.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-21T01:17:07.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-21T01:17:07.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:07.488 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:17:07.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-21T01:17:07.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-21T01:17:07.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2026-02-21T01:17:07.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:07.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:17:07.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-21T01:17:07.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-21T01:17:07.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705668 2-60129542148' 2026-02-21T01:17:07.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:07.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-21T01:17:07.721 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411331 2026-02-21T01:17:07.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411331 2026-02-21T01:17:07.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705668 2-60129542148 3-77309411331' 2026-02-21T01:17:07.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:07.722 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-21T01:17:07.722 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:07.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:17:07.723 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-21T01:17:07.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:07.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-21T01:17:07.725 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836485 2026-02-21T01:17:07.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-21T01:17:07.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:07.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-21T01:17:07.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:17:08.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:17:08.940 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:09.146 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-21T01:17:09.146 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:17:10.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:17:10.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:10.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-21T01:17:10.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:10.363 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-21T01:17:10.363 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:10.364 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:17:10.365 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-21T01:17:10.365 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:10.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-21T01:17:10.366 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 38654705668 2026-02-21T01:17:10.367 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-21T01:17:10.367 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:17:10.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705668 2026-02-21T01:17:10.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:10.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-21T01:17:10.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:10.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:17:10.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-21T01:17:10.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:10.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-21T01:17:10.587 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542148 2026-02-21T01:17:10.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-21T01:17:10.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:17:10.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-21T01:17:10.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:10.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411331 2026-02-21T01:17:10.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:10.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-21T01:17:10.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411331 2026-02-21T01:17:10.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:10.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411331 2026-02-21T01:17:10.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411331' 2026-02-21T01:17:10.813 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.3 seq 77309411331 2026-02-21T01:17:10.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T01:17:11.024 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411331 -lt 77309411331 2026-02-21T01:17:11.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:17:11.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:11.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:17:11.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:17:11.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T01:17:11.523 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:17:11.523 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:11.523 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:11.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T01:17:11.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:17:11.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:17:11.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2026-02-21T01:17:12.067 INFO:tasks.workunit.client.0.trial106.stderr:set pool 1 min_size to 3 2026-02-21T01:17:12.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2026-02-21T01:17:12.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:17:12.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:17:12.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:17:12.081 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:17:12.081 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:17:12.081 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:17:12.081 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:17:12.081 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:17:12.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:17:12.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:17:12.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:17:12.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:17:12.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:17:12.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:17:12.196 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:17:12.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:17:12.413 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:17:12.413 INFO:tasks.workunit.client.0.trial106.stderr:2 2026-02-21T01:17:12.413 INFO:tasks.workunit.client.0.trial106.stderr:3' 2026-02-21T01:17:12.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:17:12.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:12.414 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:17:12.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-21T01:17:12.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-21T01:17:12.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-21T01:17:12.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:12.490 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:17:12.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-21T01:17:12.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-21T01:17:12.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705671' 2026-02-21T01:17:12.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:12.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:17:12.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542151 2026-02-21T01:17:12.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542151 2026-02-21T01:17:12.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705671 2-60129542151' 2026-02-21T01:17:12.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:12.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-21T01:17:12.720 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411334 2026-02-21T01:17:12.721 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411334 2026-02-21T01:17:12.721 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-38654705671 2-60129542151 3-77309411334' 2026-02-21T01:17:12.721 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:12.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-21T01:17:12.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:12.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:17:12.722 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-21T01:17:12.723 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:12.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-21T01:17:12.724 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836488 2026-02-21T01:17:12.724 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-21T01:17:12.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:12.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836488 2026-02-21T01:17:12.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:17:13.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:17:13.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:14.148 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-21T01:17:14.148 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:14.149 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-21T01:17:14.149 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:14.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:17:14.151 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-21T01:17:14.151 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:14.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-21T01:17:14.153 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 38654705671 2026-02-21T01:17:14.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-21T01:17:14.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:17:14.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705671 2026-02-21T01:17:14.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:14.369 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542151 2026-02-21T01:17:14.370 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:14.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:17:14.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542151 2026-02-21T01:17:14.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:14.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542151 2026-02-21T01:17:14.373 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542151 2026-02-21T01:17:14.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-21T01:17:14.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:17:14.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-21T01:17:14.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:14.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411334 2026-02-21T01:17:14.589 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:14.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-21T01:17:14.590 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411334 2026-02-21T01:17:14.590 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:14.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411334 2026-02-21T01:17:14.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411334' 2026-02-21T01:17:14.592 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.3 seq 77309411334 2026-02-21T01:17:14.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T01:17:14.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411334 -lt 77309411334 2026-02-21T01:17:14.807 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:17:14.807 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:14.807 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:17:15.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:17:15.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:17:15.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T01:17:15.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:17:15.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:15.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:15.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T01:17:15.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:17:15.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:17:15.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-21T01:17:15.767 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:15.776 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2026-02-21T01:17:15.988 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:15.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2026-02-21T01:17:16.203 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:16.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2026-02-21T01:17:16.432 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:16.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-21T01:17:16.609 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:16.609 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-21T01:17:16.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-21T01:17:16.768 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:16.768 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-21T01:17:16.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-21T01:17:16.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2026-02-21T01:17:16.982 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:16.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2026-02-21T01:17:16.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2026-02-21T01:17:17.196 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:17.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2026-02-21T01:17:17.504 INFO:tasks.workunit.client.0.trial106.stderr:set pool 1 min_size to 2 2026-02-21T01:17:17.520 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2026-02-21T01:17:17.520 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:17:17.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:17:17.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:17:17.521 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:17:17.522 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:17:17.522 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:17:17.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:17:17.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:17:17.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:17:17.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:17:17.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:17:17.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:17:17.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:17:17.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:17:17.632 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr:2 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr:3' 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:17.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:17:17.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-21T01:17:17.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-21T01:17:17.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-21T01:17:17.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:17.921 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:17:18.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705675 2026-02-21T01:17:18.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705675 2026-02-21T01:17:18.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675' 2026-02-21T01:17:18.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:18.000 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:17:18.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-21T01:17:18.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-21T01:17:18.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-60129542154' 2026-02-21T01:17:18.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:18.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-21T01:17:18.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411337 2026-02-21T01:17:18.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411337 2026-02-21T01:17:18.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-60129542154 3-77309411337' 2026-02-21T01:17:18.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:18.155 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-21T01:17:18.155 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:18.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:17:18.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-21T01:17:18.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:18.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-21T01:17:18.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-21T01:17:18.158 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836491 2026-02-21T01:17:18.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:18.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-21T01:17:18.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:17:19.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:17:19.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:17:19.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-21T01:17:19.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:19.583 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705675 2026-02-21T01:17:19.583 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:19.584 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:17:19.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705675 2026-02-21T01:17:19.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:19.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705675 2026-02-21T01:17:19.586 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 38654705675 2026-02-21T01:17:19.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2026-02-21T01:17:19.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:17:19.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705675 -lt 38654705675 2026-02-21T01:17:19.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:19.802 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-21T01:17:19.802 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:19.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:17:19.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-21T01:17:19.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:19.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-21T01:17:19.805 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542154 2026-02-21T01:17:19.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-21T01:17:19.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:17:20.013 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-21T01:17:20.013 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:20.013 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411337 2026-02-21T01:17:20.014 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:20.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-21T01:17:20.015 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411337 2026-02-21T01:17:20.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:20.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411337 2026-02-21T01:17:20.017 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.3 seq 77309411337 2026-02-21T01:17:20.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411337' 2026-02-21T01:17:20.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T01:17:20.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411337 -lt 77309411337 2026-02-21T01:17:20.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:17:20.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:20.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:20.506 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T01:17:20.506 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:17:20.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:17:20.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:17:20.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:17:20.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:17:20.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:17:20.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:17:20.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T01:17:20.711 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:17:20.712 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:20.712 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:20.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T01:17:20.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:17:20.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:17:20.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-21T01:17:21.184 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:21.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-21T01:17:21.409 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:21.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-21T01:17:21.586 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:21.587 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-21T01:17:21.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-21T01:17:21.743 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:21.744 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-21T01:17:21.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-21T01:17:21.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2026-02-21T01:17:21.961 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:21.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2026-02-21T01:17:21.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2026-02-21T01:17:22.177 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:22.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2026-02-21T01:17:22.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2026-02-21T01:17:22.390 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:22.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2026-02-21T01:17:22.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2026-02-21T01:17:22.600 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:22.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2026-02-21T01:17:22.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:22.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:22.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:22.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:22.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:22.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:22.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2026-02-21T01:17:23.111 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already down. 2026-02-21T01:17:23.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2026-02-21T01:17:23.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1731: wait_for_peered: local cmd= 2026-02-21T01:17:23.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1732: wait_for_peered: local num_peered=-1 2026-02-21T01:17:23.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1733: wait_for_peered: local cur_peered 2026-02-21T01:17:23.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: get_timeout_delays 90 .1 2026-02-21T01:17:23.121 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:17:23.121 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:17:23.122 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:17:23.122 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:17:23.122 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:17:23.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:17:23.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: local -a delays 2026-02-21T01:17:23.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1735: wait_for_peered: local -i loop=0 2026-02-21T01:17:23.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: wait_for_peered: flush_pg_stats 2026-02-21T01:17:23.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:17:23.229 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:17:23.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:17:23.448 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:17:23.449 INFO:tasks.workunit.client.0.trial106.stderr:2 2026-02-21T01:17:23.449 INFO:tasks.workunit.client.0.trial106.stderr:3' 2026-02-21T01:17:23.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:17:23.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:23.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:17:23.503 INFO:tasks.workunit.client.0.trial106.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-21T01:17:23.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-21T01:17:23.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-21T01:17:23.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2261: flush_pg_stats: continue 2026-02-21T01:17:23.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:23.505 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:17:23.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705678 2026-02-21T01:17:23.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705678 2026-02-21T01:17:23.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 1-38654705678' 2026-02-21T01:17:23.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:23.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:17:23.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-21T01:17:23.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-21T01:17:23.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 1-38654705678 2-60129542157' 2026-02-21T01:17:23.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:17:23.661 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-21T01:17:23.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411340 2026-02-21T01:17:23.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411340 2026-02-21T01:17:23.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 1-38654705678 2-60129542157 3-77309411340' 2026-02-21T01:17:23.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:23.737 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705678 2026-02-21T01:17:23.737 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:23.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:17:23.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705678 2026-02-21T01:17:23.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:23.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705678 2026-02-21T01:17:23.741 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 38654705678 2026-02-21T01:17:23.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705678' 2026-02-21T01:17:23.741 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:17:23.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705676 -lt 38654705678 2026-02-21T01:17:23.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:17:24.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:17:24.963 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:17:25.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705678 -lt 38654705678 2026-02-21T01:17:25.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:25.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-21T01:17:25.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:25.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:17:25.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-21T01:17:25.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:25.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-21T01:17:25.172 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542157 2026-02-21T01:17:25.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-21T01:17:25.173 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:17:25.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542157 -lt 60129542157 2026-02-21T01:17:25.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:17:25.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411340 2026-02-21T01:17:25.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:17:25.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-21T01:17:25.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411340 2026-02-21T01:17:25.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:17:25.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411340 2026-02-21T01:17:25.393 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.3 seq 77309411340 2026-02-21T01:17:25.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411340' 2026-02-21T01:17:25.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T01:17:25.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411340 -lt 77309411340 2026-02-21T01:17:25.611 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: get_num_pgs 2026-02-21T01:17:25.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:25.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: test 32 == 0 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: wait_for_peered: true 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: get_num_active_or_peered 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-21T01:17:25.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-21T01:17:25.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-21T01:17:26.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: cur_peered=32 2026-02-21T01:17:26.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: get_num_pgs 2026-02-21T01:17:26.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:17:26.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:17:26.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: test 32 = 32 2026-02-21T01:17:26.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: break 2026-02-21T01:17:26.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_peered: return 0 2026-02-21T01:17:26.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-21T01:17:26.564 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2026-02-21T01:17:26.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-21T01:17:26.801 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:26.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-21T01:17:26.978 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:26.979 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-21T01:17:26.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-21T01:17:27.134 INFO:tasks.workunit.client.0.trial106.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T01:17:27.134 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-21T01:17:27.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:27.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:27.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:27.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:27.139 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:27.139 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:27.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:27.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:27.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:27.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:27.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:27.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:27.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:27.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:27.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:27.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:27.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:27.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:27.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:27.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:27.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:27.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:27.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:27.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:27.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:27.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:27.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:27.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:27.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:27.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:27.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:27.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:27.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:27.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:27.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:27.484 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:27.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:27.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:27.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:27.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:27.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:27.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:27.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:27.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:27.489 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:27.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:27.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:27.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:27.493 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:27.493 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:27.493 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:27.493 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:27.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:27.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:27.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:17:27.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:17:27.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:27.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:27.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:17:27.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:17:27.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:27.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:27.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:17:27.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:27.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:17:27.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:17:27.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:27.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:27.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:27.520 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:27.521 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:27.521 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:27.521 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:27.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:17:27.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:27.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:17:27.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:27.552 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:27.552 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:27.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:27.553 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:27.553 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:27.553 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:27.554 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:27.554 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:27.554 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:27.554 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:27.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:27.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:27.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:27.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:27.609 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:27.610 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:27.610 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:27.610 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:27.610 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:27.610 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:17:27.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2026-02-21T01:17:27.764 INFO:tasks.workunit.client.0.trial106.stderr:invalid mon(s) specified: dne 2026-02-21T01:17:27.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2026-02-21T01:17:27.869 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a 2026-02-21T01:17:27.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2026-02-21T01:17:27.957 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-21T01:17:27.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2026-02-21T01:17:28.047 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-21T01:17:28.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2026-02-21T01:17:28.158 INFO:tasks.workunit.client.0.trial106.stderr:mon.dne does not exist 2026-02-21T01:17:28.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:28.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:28.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:28.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:28.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:28.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:28.276 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:28.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:28.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:28.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:28.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:28.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:28.283 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:28.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:28.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:28.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:28.286 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:28.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:28.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:28.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:28.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:28.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:28.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:28.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:28.291 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:28.291 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.291 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.291 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:28.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:28.295 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:28.295 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:28.295 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:28.295 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:28.295 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:28.295 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:28.296 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:28.296 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:28.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:28.297 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:28.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:28.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:28.299 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:28.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:28.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:28.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:28.301 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:28.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:28.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:28.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:28.304 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:28.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:28.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:28.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:28.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:17:28.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:17:28.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:17:28.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:17:28.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:28.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:28.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:17:28.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:28.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:17:28.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:17:28.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:28.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:28.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:28.332 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:28.332 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.332 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:28.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:17:28.359 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:28.360 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:17:28.360 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:28.360 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:28.360 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:28.361 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:28.362 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:17:28.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:28.421 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.422 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.422 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:28.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:28.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:17:28.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T01:17:28.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:28.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:28.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:17:28.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:28.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T01:17:28.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T01:17:28.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:28.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:28.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:28.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:28.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:28.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T01:17:28.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:28.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:17:28.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:28.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:17:28.526 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.527 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.528 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:17:28.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:28.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get fsid 2026-02-21T01:17:28.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:17:28.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:28.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:17:28.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:28.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:17:28.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:28.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get mon_host 2026-02-21T01:17:28.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2026-02-21T01:17:28.745 INFO:tasks.workunit.client.0.trial106.stderr:invalid mon(s) specified: dne 2026-02-21T01:17:28.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:17:28.853 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons a 2026-02-21T01:17:28.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:17:28.946 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b 2026-02-21T01:17:28.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2026-02-21T01:17:29.035 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b 2026-02-21T01:17:29.038 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:17:29.143 INFO:tasks.workunit.client.0.trial106.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T01:17:29.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:17:29.267 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:17:29.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:17:29.393 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:17:29.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2026-02-21T01:17:29.516 INFO:tasks.workunit.client.0.trial106.stderr:mon.dne does not exist 2026-02-21T01:17:29.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:29.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:29.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:29.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:29.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:29.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:29.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:29.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:29.634 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:29.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:29.639 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:29.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:29.640 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:29.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:29.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:29.642 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:29.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:29.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:29.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:29.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:29.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:29.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:29.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:29.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:29.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:29.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:29.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:29.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:29.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:29.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:29.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:29.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:29.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:29.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:29.654 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:29.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:29.655 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:29.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:29.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:29.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:29.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:29.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:29.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:29.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:29.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:29.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:29.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:29.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:29.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:29.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:29.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:29.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:17:29.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:17:29.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:17:29.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:17:29.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:17:29.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:17:29.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:17:29.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:17:29.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:29.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:17:29.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.689 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:29.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:17:29.720 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:29.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:17:29.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:29.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:29.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:29.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:29.722 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:29.723 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:17:29.777 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:17:29.777 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:29.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:29.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:29.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:29.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T01:17:29.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:29.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T01:17:29.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:29.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:17:29.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:29.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:17:29.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:29.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:29.886 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:17:29.886 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:29.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get fsid 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:17:29.943 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:17:29.944 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:17:29.945 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:29.945 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:29.945 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:29.945 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:17:29.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:29.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get mon_host 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-21T01:17:29.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-21T01:17:30.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:30.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:30.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:30.018 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:30.018 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:30.019 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:30.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:30.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-21T01:17:30.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T01:17:30.047 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T01:17:30.048 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:30.048 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:30.048 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:30.048 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.c.asok 2026-02-21T01:17:30.049 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:30.049 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.c.asok config get fsid 2026-02-21T01:17:30.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-21T01:17:30.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:30.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T01:17:30.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:30.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.c.asok 2026-02-21T01:17:30.106 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:30.107 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.c.asok config get mon_host 2026-02-21T01:17:30.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2026-02-21T01:17:30.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:17:30.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-21T01:17:30.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:17:30.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-21T01:17:30.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:17:30.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:17:30.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636710 2026-02-21T01:17:30.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:30.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636650 -lt 1771636710 ]] 2026-02-21T01:17:30.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:17:30.164 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:30.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:30.476 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:30.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:17:30.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:30.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:17:30.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:17:30.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:17:30.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:17:30.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:17:30.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2026-02-21T01:17:30.587 INFO:tasks.workunit.client.0.trial106.stderr:invalid mon(s) specified: dne 2026-02-21T01:17:30.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:17:30.710 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c) after stopping a 2026-02-21T01:17:30.718 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:17:30.833 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,c) after stopping b 2026-02-21T01:17:30.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:17:30.948 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b) after stopping c 2026-02-21T01:17:30.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2026-02-21T01:17:31.044 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a b 2026-02-21T01:17:31.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2026-02-21T01:17:31.131 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons b c 2026-02-21T01:17:31.135 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2026-02-21T01:17:31.221 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available () after stopping mons a b c 2026-02-21T01:17:31.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:17:31.323 INFO:tasks.workunit.client.0.trial106.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T01:17:31.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:17:31.447 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:17:31.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:17:31.564 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:17:31.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:17:31.685 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.c 2026-02-21T01:17:31.693 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2026-02-21T01:17:31.694 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:31.694 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:31.694 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:31.694 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:31.695 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:31.802 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:31.802 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2026-02-21T01:17:31.802 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:17:31.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2026-02-21T01:17:31.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:17:31.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2026-02-21T01:17:31.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:17:31.803 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:17:31.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636711 2026-02-21T01:17:31.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:31.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636651 -lt 1771636711 ]] 2026-02-21T01:17:31.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:31.805 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:31.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:31.919 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:31.920 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:31.920 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:31.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:31.930 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:31.930 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:31.930 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636651 -lt 1771636711 ]] 2026-02-21T01:17:31.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:31.931 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:32.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:32.042 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:32.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:32.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:32.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:32.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636652 -lt 1771636711 ]] 2026-02-21T01:17:32.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:32.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:32.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:32.170 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:32.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:32.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:32.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:32.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636652 -lt 1771636711 ]] 2026-02-21T01:17:32.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:32.183 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:32.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:32.293 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.294 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:32.294 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:32.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:32.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:32.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636652 -lt 1771636711 ]] 2026-02-21T01:17:32.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:32.306 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:32.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:32.420 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:32.421 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:32.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:32.430 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:32.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636652 -lt 1771636711 ]] 2026-02-21T01:17:32.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:32.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:32.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:32.547 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:32.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:32.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:32.557 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:32.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636652 -lt 1771636711 ]] 2026-02-21T01:17:32.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:32.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:32.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:32.671 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:32.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:32.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:32.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:32.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:32.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636652 -lt 1771636711 ]] 2026-02-21T01:17:32.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:32.683 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:35.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:35.800 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:35.801 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:35.801 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:35.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:35.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:35.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:35.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636655 -lt 1771636711 ]] 2026-02-21T01:17:35.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:35.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:38.927 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:38.927 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:38.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:38.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:38.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:38.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:38.938 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:38.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636658 -lt 1771636711 ]] 2026-02-21T01:17:38.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:38.939 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:39.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:39.053 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:39.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:39.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:39.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:39.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:39.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:39.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636659 -lt 1771636711 ]] 2026-02-21T01:17:39.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:39.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:39.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:39.175 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:39.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:39.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:39.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:39.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:39.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:39.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636659 -lt 1771636711 ]] 2026-02-21T01:17:39.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:39.188 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:39.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:39.299 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:39.299 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:39.299 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:39.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:39.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:39.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:39.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636659 -lt 1771636711 ]] 2026-02-21T01:17:39.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T01:17:39.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:46.834 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:46.834 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:46.835 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T01:17:46.835 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:29.681421Z","created":"2026-02-21T01:17:29.681421Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:46.845 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:17:46.845 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:17:46.845 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:17:46.846 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:17:46.846 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:17:46.846 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:17:46.929 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons a 2026-02-21T01:17:46.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:17:47.043 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,c) after stopping b 2026-02-21T01:17:47.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:17:47.146 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons c 2026-02-21T01:17:47.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:17:47.233 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-21T01:17:47.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:17:47.321 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-21T01:17:47.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:17:47.427 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:17:47.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:17:47.533 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-21T01:17:47.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:47.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:47.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:47.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:47.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:47.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:47.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:47.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:47.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:47.651 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:47.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:47.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:47.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:47.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:47.655 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:47.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:47.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:47.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:47.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:47.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:47.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:47.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:47.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:47.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:47.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:47.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:47.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:47.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:17:47.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:17:47.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:17:47.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:17:47.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:17:47.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:17:47.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:47.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:17:47.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:17:47.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:17:47.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:17:47.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:17:47.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:17:47.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:17:47.678 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.678 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:17:47.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:17:47.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:17:47.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:17:47.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:17:47.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:17:47.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:17:47.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:17:47.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:47.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:17:47.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:17:47.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:47.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:47.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:47.704 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:47.704 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.704 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:47.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:17:47.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:47.734 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:17:47.734 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:47.734 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:47.734 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.735 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.736 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:47.736 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:47.736 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:17:47.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:17:47.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:47.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:17:47.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.791 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.792 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:17:47.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:47.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:17:47.846 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T01:17:47.846 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:47.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:47.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:17:47.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:47.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T01:17:47.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T01:17:47.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:47.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:47.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:47.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:47.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.867 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:47.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T01:17:47.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:47.903 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:17:47.903 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:47.903 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:17:47.904 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:47.904 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:17:47.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:47.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get fsid 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:47.961 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:47.962 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:47.962 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:17:47.962 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:47.962 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get mon_host 2026-02-21T01:17:48.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-21T01:17:48.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:48.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:48.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-21T01:17:48.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:48.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-21T01:17:48.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-21T01:17:48.034 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:48.034 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:48.035 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:48.035 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:48.035 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:48.035 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:48.035 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:48.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-21T01:17:48.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:48.063 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-21T01:17:48.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:48.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T01:17:48.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:48.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:48.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T01:17:48.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T01:17:48.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T01:17:48.066 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:48.066 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:48.066 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:48.066 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.c.asok 2026-02-21T01:17:48.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:48.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.c.asok config get fsid 2026-02-21T01:17:48.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-21T01:17:48.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:48.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T01:17:48.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:48.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:48.122 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T01:17:48.122 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T01:17:48.122 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T01:17:48.122 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:48.122 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:48.123 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:48.123 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.c.asok 2026-02-21T01:17:48.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:48.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.c.asok config get mon_host 2026-02-21T01:17:48.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-21T01:17:48.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:17:48.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:17:48.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-21T01:17:48.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:17:48.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-21T01:17:48.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:48.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:17:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-21T01:17:48.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:17:48.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-21T01:17:48.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:48.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T01:17:48.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:17:48.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:17:48.227 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T01:17:48.227 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T01:17:48.227 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T01:17:48.228 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:48.228 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:48.228 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:48.228 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.d.asok 2026-02-21T01:17:48.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:48.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.d.asok config get fsid 2026-02-21T01:17:48.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-21T01:17:48.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:17:48.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T01:17:48.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:17:48.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:17:48.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T01:17:48.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T01:17:48.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T01:17:48.284 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:17:48.284 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:17:48.284 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:17:48.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.d.asok 2026-02-21T01:17:48.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:17:48.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.d.asok config get mon_host 2026-02-21T01:17:48.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2026-02-21T01:17:48.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:17:48.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-21T01:17:48.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:17:48.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-21T01:17:48.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:17:48.337 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:17:48.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636728 2026-02-21T01:17:48.338 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:48.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636668 -lt 1771636728 ]] 2026-02-21T01:17:48.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:17:48.339 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:54.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:54.451 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:54.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:17:54.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:54.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:17:54.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:17:54.461 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:54.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636674 -lt 1771636728 ]] 2026-02-21T01:17:54.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:17:54.463 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:17:57.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:17:57.581 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:57.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:17:57.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:17:57.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:17:57.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:17:57.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:17:57.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:17:57.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:17:57.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2026-02-21T01:17:58.303 INFO:tasks.workunit.client.0.trial106.stderr:invalid mon(s) specified: dne 2026-02-21T01:17:58.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:17:58.417 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d) after stopping a 2026-02-21T01:17:58.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:17:58.533 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,c,d) after stopping b 2026-02-21T01:17:58.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:17:58.643 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b,d) after stopping c 2026-02-21T01:17:58.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-21T01:17:58.753 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b,c) after stopping d 2026-02-21T01:17:58.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2026-02-21T01:17:58.843 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons a b 2026-02-21T01:17:58.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2026-02-21T01:17:58.934 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons c d 2026-02-21T01:17:58.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:17:59.039 INFO:tasks.workunit.client.0.trial106.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T01:17:59.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:17:59.166 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:17:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:17:59.279 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:17:59.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:17:59.404 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.c 2026-02-21T01:17:59.412 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-21T01:17:59.413 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:17:59.413 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:17:59.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:17:59.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:17:59.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:17:59.521 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:17:59.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636739 2026-02-21T01:17:59.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:17:59.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636679 -lt 1771636739 ]] 2026-02-21T01:17:59.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:17:59.523 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:02.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:02.638 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:02.638 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:02.638 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:02.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:02.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:02.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:02.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636682 -lt 1771636739 ]] 2026-02-21T01:18:02.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:18:02.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:02.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:02.766 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:02.767 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:02.767 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:02.776 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:02.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:02.777 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:02.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636682 -lt 1771636739 ]] 2026-02-21T01:18:02.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:18:02.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:05.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:05.889 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:05.889 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:05.889 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:05.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:05.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:05.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:05.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636685 -lt 1771636739 ]] 2026-02-21T01:18:05.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:18:05.902 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:06.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:06.015 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:06.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:06.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:06.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:06.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:06.026 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:06.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636686 -lt 1771636739 ]] 2026-02-21T01:18:06.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:18:06.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:09.140 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:09.141 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:09.141 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:09.141 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:09.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:09.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:09.150 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:09.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636689 -lt 1771636739 ]] 2026-02-21T01:18:09.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:18:09.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:09.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:09.268 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:09.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:09.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:09.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:09.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:09.279 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:09.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636689 -lt 1771636739 ]] 2026-02-21T01:18:09.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:18:09.280 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:14.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:14.416 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:14.417 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:18:14.417 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:17:47.696270Z","created":"2026-02-21T01:17:47.696270Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:14.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:18:14.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:18:14.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:18:14.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:18:14.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:18:14.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:18:14.528 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d) after stopping a 2026-02-21T01:18:14.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:18:14.634 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2026-02-21T01:18:14.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:18:14.723 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2026-02-21T01:18:14.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-21T01:18:14.812 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2026-02-21T01:18:14.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:18:14.917 INFO:tasks.workunit.client.0.trial106.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T01:18:14.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:18:15.045 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:18:15.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:18:15.158 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:18:15.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:18:18.282 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.c 2026-02-21T01:18:18.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2026-02-21T01:18:21.403 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.d 2026-02-21T01:18:21.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:18:21.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:18:21.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:18:21.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:18:21.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:18:21.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:18:21.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:18:21.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:18:21.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:18:21.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:18:21.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:18:21.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:18:21.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:18:21.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:18:21.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:18:21.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:18:21.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:18:21.532 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:18:21.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:18:21.533 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:18:21.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:18:21.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:18:21.535 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:18:21.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:18:21.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:18:21.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:18:21.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:18:21.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:18:21.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:18:21.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:18:21.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:18:21.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:18:21.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:18:21.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:18:21.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:18:21.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:18:21.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:18:21.552 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:18:21.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:18:21.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:18:21.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:18:21.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:18:21.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:18:21.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:18:21.556 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:18:21.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:18:21.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:18:21.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:18:21.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:18:21.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:18:21.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:18:21.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:18:21.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T01:18:21.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:18:21.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.39862 2026-02-21T01:18:21.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:18:21.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:18:21.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:18:21.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T01:18:21.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:18:21.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T01:18:21.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:18:21.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:18:21.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:21.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:18:21.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:18:21.617 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:21.618 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get fsid 2026-02-21T01:18:21.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:18:21.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:21.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:18:21.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:18:21.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.a.asok 2026-02-21T01:18:21.675 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:21.675 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.a.asok config get mon_host 2026-02-21T01:18:21.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T01:18:21.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:18:21.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:18:21.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:18:21.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:18:21.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T01:18:21.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T01:18:21.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:18:21.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:18:21.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:18:21.751 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:18:21.752 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.752 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.752 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:18:21.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T01:18:21.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:18:21.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:18:21.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:21.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:18:21.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:18:21.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.785 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:18:21.786 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:21.786 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get fsid 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:18:21.842 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.b.asok 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:21.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.b.asok config get mon_host 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-21T01:18:21.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-21T01:18:21.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:18:21.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:18:21.918 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:18:21.919 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:18:21.919 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.919 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.919 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:18:21.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-21T01:18:21.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:18:21.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-21T01:18:21.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:21.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T01:18:21.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:18:21.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:18:21.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T01:18:21.949 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T01:18:21.949 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T01:18:21.949 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:21.949 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:21.949 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:21.949 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.c.asok 2026-02-21T01:18:21.950 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:21.950 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.c.asok config get fsid 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T01:18:22.008 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T01:18:22.009 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:22.009 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.009 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.009 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.c.asok 2026-02-21T01:18:22.009 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:22.009 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.c.asok config get mon_host 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-21T01:18:22.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:18:22.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-21T01:18:22.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:18:22.116 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-21T01:18:22.116 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:22.116 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T01:18:22.116 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.117 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.118 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.d.asok 2026-02-21T01:18:22.118 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:22.118 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.d.asok config get fsid 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T01:18:22.176 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T01:18:22.177 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:22.178 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.178 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.178 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.d.asok 2026-02-21T01:18:22.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:22.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.d.asok config get mon_host 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/e 2026-02-21T01:18:22.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2026-02-21T01:18:22.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:18:22.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:18:22.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:18:22.254 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:18:22.254 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.255 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.39862/$cluster-$name.asok' 2026-02-21T01:18:22.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.39862/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2026-02-21T01:18:22.281 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:18:22.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2026-02-21T01:18:22.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:22.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-21T01:18:22.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:18:22.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.e.asok 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:22.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.e.asok config get fsid 2026-02-21T01:18:22.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2026-02-21T01:18:22.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:18:22.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-21T01:18:22.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:18:22.341 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:18:22.342 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.39862/ceph-mon.e.asok 2026-02-21T01:18:22.343 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:18:22.343 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.39862/ceph-mon.e.asok config get mon_host 2026-02-21T01:18:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2026-02-21T01:18:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:18:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2026-02-21T01:18:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:18:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2026-02-21T01:18:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:18:22.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:18:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636762 2026-02-21T01:18:22.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636702 -lt 1771636762 ]] 2026-02-21T01:18:22.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-21T01:18:22.402 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:28.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:28.519 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:28.519 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2026-02-21T01:18:28.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:28.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:18:28.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:18:28.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:18:28.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:18:28.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:18:28.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2026-02-21T01:18:28.633 INFO:tasks.workunit.client.0.trial106.stderr:invalid mon(s) specified: dne 2026-02-21T01:18:28.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:18:28.746 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d,e) after stopping a 2026-02-21T01:18:28.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:18:28.872 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,c,d,e) after stopping b 2026-02-21T01:18:28.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:18:28.985 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b,d,e) after stopping c 2026-02-21T01:18:28.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-21T01:18:29.112 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b,c,e) after stopping d 2026-02-21T01:18:29.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-21T01:18:32.260 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b,c,d) after stopping e 2026-02-21T01:18:32.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2026-02-21T01:18:32.385 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (c,d,e) after stopping a b 2026-02-21T01:18:32.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2026-02-21T01:18:32.502 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (a,b,e) after stopping c d 2026-02-21T01:18:32.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2026-02-21T01:18:32.609 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons a b c 2026-02-21T01:18:32.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:18:32.719 INFO:tasks.workunit.client.0.trial106.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T01:18:32.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:18:32.844 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:18:32.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:18:32.961 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:18:32.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:18:33.088 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.c 2026-02-21T01:18:33.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-21T01:18:33.204 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.d 2026-02-21T01:18:33.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-21T01:18:33.331 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.e 2026-02-21T01:18:33.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-21T01:18:33.343 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:18:33.343 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:18:33.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:18:33.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:18:33.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:18:33.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:18:33.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2026-02-21T01:18:33.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:18:33.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-21T01:18:33.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:18:33.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-21T01:18:33.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:18:33.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:18:33.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636773 2026-02-21T01:18:33.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:33.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636713 -lt 1771636773 ]] 2026-02-21T01:18:33.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:33.454 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:33.563 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:33.564 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:33.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:33.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:33.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:33.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636713 -lt 1771636773 ]] 2026-02-21T01:18:33.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:33.576 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:33.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:33.689 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:33.700 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:33.700 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:33.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:33.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636713 -lt 1771636773 ]] 2026-02-21T01:18:33.702 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:33.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:33.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:33.815 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:33.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:33.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:33.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:33.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636713 -lt 1771636773 ]] 2026-02-21T01:18:33.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:33.826 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:33.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:33.944 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:33.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:33.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:33.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:33.954 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:33.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636713 -lt 1771636773 ]] 2026-02-21T01:18:33.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:33.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:34.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:34.071 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:34.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:34.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:34.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:34.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:34.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:34.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636714 -lt 1771636773 ]] 2026-02-21T01:18:34.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:34.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:34.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:34.200 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:34.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:34.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:34.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:34.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:34.210 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:34.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636714 -lt 1771636773 ]] 2026-02-21T01:18:34.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:34.212 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:34.322 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:34.322 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:34.323 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:34.323 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:34.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:34.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:34.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:34.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636714 -lt 1771636773 ]] 2026-02-21T01:18:34.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:34.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:37.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:37.453 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:37.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:37.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:37.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:37.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636717 -lt 1771636773 ]] 2026-02-21T01:18:37.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:37.464 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:37.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:37.574 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:37.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:37.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:37.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:37.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636717 -lt 1771636773 ]] 2026-02-21T01:18:37.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:37.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:37.700 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:37.700 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:37.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:37.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:37.710 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:37.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636717 -lt 1771636773 ]] 2026-02-21T01:18:37.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:37.712 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:37.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:37.829 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.830 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.830 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:37.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:37.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:37.841 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:37.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636717 -lt 1771636773 ]] 2026-02-21T01:18:37.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:37.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:37.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:37.952 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:37.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:37.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:37.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:37.964 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:37.965 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636717 -lt 1771636773 ]] 2026-02-21T01:18:37.965 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:37.965 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:38.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:38.081 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:38.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:38.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:38.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:38.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:38.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:38.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636718 -lt 1771636773 ]] 2026-02-21T01:18:38.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:38.094 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:38.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:38.207 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:38.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:38.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:38.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:38.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:38.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:38.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636718 -lt 1771636773 ]] 2026-02-21T01:18:38.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:38.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:41.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:41.336 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:41.345 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:41.345 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:41.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:41.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636721 -lt 1771636773 ]] 2026-02-21T01:18:41.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:41.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:41.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:41.461 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:41.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:41.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:41.472 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:41.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636721 -lt 1771636773 ]] 2026-02-21T01:18:41.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:41.474 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:41.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:41.586 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:41.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:41.597 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:41.597 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:41.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636721 -lt 1771636773 ]] 2026-02-21T01:18:41.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:41.598 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:41.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:41.714 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:41.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:41.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:41.725 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:41.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636721 -lt 1771636773 ]] 2026-02-21T01:18:41.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:41.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:41.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:41.841 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:41.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.852 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:41.852 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:41.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:41.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636721 -lt 1771636773 ]] 2026-02-21T01:18:41.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:41.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:41.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:41.972 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:41.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:41.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:41.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:41.982 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:41.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636721 -lt 1771636773 ]] 2026-02-21T01:18:41.983 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:41.983 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.102 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.102 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.102 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.223 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.233 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.349 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.349 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.350 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.350 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.359 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.359 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.359 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.361 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.361 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.479 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.479 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.480 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.480 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.490 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.491 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.603 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.614 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.729 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.729 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.730 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.738 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.740 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.856 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.856 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.857 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.857 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.868 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:42.983 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:42.984 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:42.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:42.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:42.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:42.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:42.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636722 -lt 1771636773 ]] 2026-02-21T01:18:42.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:42.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:43.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:43.106 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:43.107 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:43.107 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:43.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:43.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:43.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:43.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636723 -lt 1771636773 ]] 2026-02-21T01:18:43.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:43.119 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:43.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:43.234 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:43.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:43.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:43.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:18:43.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:18:43.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:18:43.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636723 -lt 1771636773 ]] 2026-02-21T01:18:43.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T01:18:43.246 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:18:53.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:18:53.382 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:53.383 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T01:18:53.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:18:53.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:18:53.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:18:53.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:18:53.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:18:53.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:18:53.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:18:56.496 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d,e) after stopping a 2026-02-21T01:18:56.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:18:56.615 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (c,d,e) after stopping b 2026-02-21T01:18:56.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:18:56.738 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,d,e) after stopping c 2026-02-21T01:18:56.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-21T01:18:56.851 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,e) after stopping d 2026-02-21T01:18:56.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-21T01:18:59.997 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d) after stopping e 2026-02-21T01:19:00.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:19:00.113 INFO:tasks.workunit.client.0.trial106.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T01:19:00.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:19:00.226 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:19:00.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:19:00.343 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.b 2026-02-21T01:19:00.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:19:00.455 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.c 2026-02-21T01:19:00.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-21T01:19:00.571 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.d 2026-02-21T01:19:00.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-21T01:19:00.695 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.e 2026-02-21T01:19:00.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2026-02-21T01:19:00.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:00.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:00.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:00.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:00.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:00.810 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:00.810 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2026-02-21T01:19:00.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T01:19:00.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-21T01:19:00.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T01:19:00.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-21T01:19:00.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T01:19:00.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T01:19:00.812 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771636800 2026-02-21T01:19:00.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636740 -lt 1771636800 ]] 2026-02-21T01:19:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:00.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:00.928 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:00.928 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:00.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:00.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:00.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:00.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:00.938 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:00.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636740 -lt 1771636800 ]] 2026-02-21T01:19:00.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:00.940 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:01.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:01.052 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:01.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:01.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:01.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:01.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:01.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:01.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636741 -lt 1771636800 ]] 2026-02-21T01:19:01.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:01.063 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:04.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:04.180 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:04.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:04.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:04.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:04.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:04.190 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:04.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636744 -lt 1771636800 ]] 2026-02-21T01:19:04.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:04.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:07.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:07.328 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:07.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:07.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:07.338 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:07.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636747 -lt 1771636800 ]] 2026-02-21T01:19:07.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:07.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:07.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:07.452 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:07.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:07.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:07.463 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:07.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636747 -lt 1771636800 ]] 2026-02-21T01:19:07.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:07.464 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:07.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:07.580 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.581 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.581 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:07.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:07.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:07.591 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636747 -lt 1771636800 ]] 2026-02-21T01:19:07.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:07.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:07.702 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:07.703 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:07.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:07.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:07.714 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:07.715 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636747 -lt 1771636800 ]] 2026-02-21T01:19:07.715 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:07.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:07.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:07.831 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.831 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.831 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:07.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:07.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:07.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:07.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636747 -lt 1771636800 ]] 2026-02-21T01:19:07.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:07.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:07.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:07.957 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.957 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:07.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:07.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:07.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:07.969 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:07.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636747 -lt 1771636800 ]] 2026-02-21T01:19:07.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:07.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:08.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:08.082 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:08.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:08.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:08.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:08.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636748 -lt 1771636800 ]] 2026-02-21T01:19:08.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:08.095 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:08.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:08.212 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.212 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:08.212 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:08.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:08.221 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:08.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636748 -lt 1771636800 ]] 2026-02-21T01:19:08.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:08.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:08.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:08.339 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:08.349 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:08.349 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:08.350 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:08.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636748 -lt 1771636800 ]] 2026-02-21T01:19:08.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:08.351 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:08.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:08.462 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.463 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:08.463 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:08.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T01:19:08.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T01:19:08.474 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T01:19:08.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771636748 -lt 1771636800 ]] 2026-02-21T01:19:08.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T01:19:08.475 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T01:19:16.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T01:19:16.006 INFO:tasks.workunit.client.0.trial106.stderr:{"election_epoch":16,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:16.006 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T01:19:16.006 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":16,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"b78e5e8e-e5e4-45ad-83b1-0cf2c342d6b7","modified":"2026-02-21T01:18:21.577351Z","created":"2026-02-21T01:18:21.577351Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-21T01:19:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T01:19:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T01:19:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T01:19:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T01:19:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T01:19:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-21T01:19:16.121 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d) after stopping a 2026-02-21T01:19:16.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-21T01:19:19.216 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons b 2026-02-21T01:19:19.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-21T01:19:22.309 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons c 2026-02-21T01:19:22.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-21T01:19:22.399 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons d 2026-02-21T01:19:22.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-21T01:19:25.509 INFO:tasks.workunit.client.0.trial106.stderr:quorum should be preserved (b,c,d) after stopping e 2026-02-21T01:19:25.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-21T01:19:25.615 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-21T01:19:25.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-21T01:19:25.726 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.a 2026-02-21T01:19:25.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-21T01:19:25.820 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: removing mon.b would break quorum 2026-02-21T01:19:25.823 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-21T01:19:25.909 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-21T01:19:25.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-21T01:19:29.000 INFO:tasks.workunit.client.0.trial106.stderr:Error EBUSY: removing mon.d would break quorum 2026-02-21T01:19:29.003 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-21T01:19:29.106 INFO:tasks.workunit.client.0.trial106.stderr:safe to remove mon.e 2026-02-21T01:19:29.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T01:19:29.118 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:29.118 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:29.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:29.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:29.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T01:19:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:19:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:19:29.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:29.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:29.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:29.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:29.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:29.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:29.233 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:29.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:29.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:29.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:29.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:29.236 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:29.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:29.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:29.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:29.238 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:29.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:29.240 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:29.240 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:19:29.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:29.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:19:29.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/ok-to-stop 0 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T01:19:29.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T01:19:29.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:29.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:29.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:29.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:29.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:29.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:29.261 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:29.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:29.262 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:29.263 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:29.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:29.264 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:29.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:29.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:29.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:29.266 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:29.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:29.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T01:19:29.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T01:19:29.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:29.270 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.270 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.39862 2026-02-21T01:19:29.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.39862 2026-02-21T01:19:29.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:29.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:29.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-21T01:19:29.272 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:19:29.273 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:19:29.321 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2026-02-21T01:19:29.322 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2026-02-21T01:19:29.374 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-21T01:19:29.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/rados-striper 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none ' 2026-02-21T01:19:29.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/rados-striper 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/rados-striper 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:29.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:29.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:29.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:29.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:29.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:29.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:29.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:29.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:29.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:29.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:29.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:29.386 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:29.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:29.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:29.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-21T01:19:29.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:29.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:29.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55911 2026-02-21T01:19:29.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:29.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:29.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/rados-striper 2026-02-21T01:19:29.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:19:29.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:29.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55911 2026-02-21T01:19:29.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/rados-striper 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/rados-striper/a 2026-02-21T01:19:29.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:29.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:29.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:19:29.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:19:29.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:19:29.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:29.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:29.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55911/ceph-mon.a.asok 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:29.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55911/ceph-mon.a.asok config get fsid 2026-02-21T01:19:29.510 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:19:29.510 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:29.510 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:29.510 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:19:29.510 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:19:29.510 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55911/ceph-mon.a.asok 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:29.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55911/ceph-mon.a.asok config get mon_host 2026-02-21T01:19:29.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2026-02-21T01:19:29.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-21T01:19:29.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:19:29.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:19:29.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/0 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-21T01:19:29.568 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-21T01:19:29.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:29.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:29.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:29.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:29.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:19:29.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:19:29.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/0 2026-02-21T01:19:29.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:19:29.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=698abee6-862f-440c-b72a-22b94ddd6ac2 2026-02-21T01:19:29.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 698abee6-862f-440c-b72a-22b94ddd6ac2' 2026-02-21T01:19:29.571 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 698abee6-862f-440c-b72a-22b94ddd6ac2 2026-02-21T01:19:29.572 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:19:29.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhCJlpupd8IhAAEa+ax9WOY3cTDO602DsKyg== 2026-02-21T01:19:29.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhCJlpupd8IhAAEa+ax9WOY3cTDO602DsKyg=="}' 2026-02-21T01:19:29.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 698abee6-862f-440c-b72a-22b94ddd6ac2 -i td/rados-striper/0/new.json 2026-02-21T01:19:29.698 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:29.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/0/new.json 2026-02-21T01:19:29.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhCJlpupd8IhAAEa+ax9WOY3cTDO602DsKyg== --osd-uuid 698abee6-862f-440c-b72a-22b94ddd6ac2 2026-02-21T01:19:29.722 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:29.720+0000 7f2d6367fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:29.723 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:29.720+0000 7f2d6367fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:29.724 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:29.720+0000 7f2d6367fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:29.725 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:29.724+0000 7f2d6367fa40 -1 bdev(0x556a26bbac00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:19:29.725 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:29.724+0000 7f2d6367fa40 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2026-02-21T01:19:30.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/0/keyring 2026-02-21T01:19:30.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:19:30.374 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:19:30.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:19:30.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:19:30.499 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:30.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:19:30.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:19:30.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:19:30.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:19:30.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:19:30.521 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:30.516+0000 7f0e797c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:30.523 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:30.520+0000 7f0e797c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:30.524 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:30.520+0000 7f0e797c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:30.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:19:30.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:30.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:30.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:30.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:30.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:30.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:30.608 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:30.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:30.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:30.718 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:30.867 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:30.864+0000 7f0e797c8a40 -1 Falling back to public interface 2026-02-21T01:19:31.467 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:31.464+0000 7f0e797c8a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:19:31.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:31.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:31.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:31.719 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:31.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:31.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:31.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:32.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:32.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:32.827 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:32.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:19:32.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:32.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:32.939 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1741804183,v1:127.0.0.1:6801/1741804183] [v2:127.0.0.1:6802/1741804183,v1:127.0.0.1:6803/1741804183] exists,up 698abee6-862f-440c-b72a-22b94ddd6ac2 2026-02-21T01:19:32.942 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/1 2026-02-21T01:19:32.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:32.944 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:32.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:32.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:32.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:19:32.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:19:32.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/1 2026-02-21T01:19:32.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:19:32.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2cc00fa3-dedc-41d9-8ca0-a01e1efb9336 2026-02-21T01:19:32.947 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 2cc00fa3-dedc-41d9-8ca0-a01e1efb9336 2026-02-21T01:19:32.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2cc00fa3-dedc-41d9-8ca0-a01e1efb9336' 2026-02-21T01:19:32.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:19:32.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkCJlp44rTOBAAOl0axTJKlUyPDx1gZAXEEQ== 2026-02-21T01:19:32.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkCJlp44rTOBAAOl0axTJKlUyPDx1gZAXEEQ=="}' 2026-02-21T01:19:32.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2cc00fa3-dedc-41d9-8ca0-a01e1efb9336 -i td/rados-striper/1/new.json 2026-02-21T01:19:33.074 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:33.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/1/new.json 2026-02-21T01:19:33.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkCJlp44rTOBAAOl0axTJKlUyPDx1gZAXEEQ== --osd-uuid 2cc00fa3-dedc-41d9-8ca0-a01e1efb9336 2026-02-21T01:19:33.098 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.096+0000 7f54fd0cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:33.100 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.096+0000 7f54fd0cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:33.101 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.100+0000 7f54fd0cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:33.101 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.100+0000 7f54fd0cea40 -1 bdev(0x55e77bdb4c00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:19:33.101 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.100+0000 7f54fd0cea40 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2026-02-21T01:19:33.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/1/keyring 2026-02-21T01:19:33.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:19:33.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:19:33.742 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:19:33.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:19:33.863 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:19:33.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:19:33.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:19:33.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:19:33.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:19:33.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:19:33.885 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.884+0000 7fcd2f7e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:33.887 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.884+0000 7fcd2f7e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:33.888 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:33.884+0000 7fcd2f7e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:33.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:19:33.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:33.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:19:33.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:33.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:33.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:33.972 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:33.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:33.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:33.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:19:34.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:34.230 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:34.228+0000 7fcd2f7e5a40 -1 Falling back to public interface 2026-02-21T01:19:34.847 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:34.844+0000 7fcd2f7e5a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:19:35.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:35.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:35.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:35.083 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:35.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:19:35.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:35.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:36.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:36.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:36.197 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:36.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:19:36.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:36.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:19:36.308 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1435050016,v1:127.0.0.1:6809/1435050016] [v2:127.0.0.1:6810/1435050016,v1:127.0.0.1:6811/1435050016] exists,up 2cc00fa3-dedc-41d9-8ca0-a01e1efb9336 2026-02-21T01:19:36.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:36.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:36.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:36.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2026-02-21T01:19:36.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/2 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:36.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:19:36.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:19:36.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/2 2026-02-21T01:19:36.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:19:36.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=92f90b8c-b32d-4cea-886e-4715df563841 2026-02-21T01:19:36.315 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 92f90b8c-b32d-4cea-886e-4715df563841 2026-02-21T01:19:36.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 92f90b8c-b32d-4cea-886e-4715df563841' 2026-02-21T01:19:36.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:19:36.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoCJlpilBlExAABiwXwEwU8TGnIzDj/87ufw== 2026-02-21T01:19:36.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoCJlpilBlExAABiwXwEwU8TGnIzDj/87ufw=="}' 2026-02-21T01:19:36.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 92f90b8c-b32d-4cea-886e-4715df563841 -i td/rados-striper/2/new.json 2026-02-21T01:19:36.443 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:36.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/2/new.json 2026-02-21T01:19:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoCJlpilBlExAABiwXwEwU8TGnIzDj/87ufw== --osd-uuid 92f90b8c-b32d-4cea-886e-4715df563841 2026-02-21T01:19:36.479 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:36.476+0000 7f3c4bc03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:36.480 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:36.476+0000 7f3c4bc03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:36.481 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:36.480+0000 7f3c4bc03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:36.482 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:36.480+0000 7f3c4bc03a40 -1 bdev(0x55aee55f6c00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:19:36.482 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:36.480+0000 7f3c4bc03a40 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2026-02-21T01:19:37.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/2/keyring 2026-02-21T01:19:37.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:19:37.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:19:37.122 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:19:37.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:19:37.244 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:19:37.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:19:37.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=4e6cb4db-0b78-4304-b7cb-bdfc03017bf1 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.55911/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:19:37.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:19:37.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:19:37.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:19:37.266 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:37.264+0000 7efcc48efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:37.268 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:37.264+0000 7efcc48efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:37.269 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:37.264+0000 7efcc48efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:37.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:19:37.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:37.623 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:37.620+0000 7efcc48efa40 -1 Falling back to public interface 2026-02-21T01:19:38.279 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:38.276+0000 7efcc48efa40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:19:38.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:38.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:38.464 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:38.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:38.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:38.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:19:38.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:39.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:39.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:39.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:19:39.570 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:39.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:39.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:19:39.682 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/3555035831,v1:127.0.0.1:6817/3555035831] [v2:127.0.0.1:6818/3555035831,v1:127.0.0.1:6819/3555035831] exists,up 92f90b8c-b32d-4cea-886e-4715df563841 2026-02-21T01:19:39.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:39.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:39.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:39.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2026-02-21T01:19:39.683 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:19:39.787 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:19:39.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:19:39.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:19:39.928 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:19:39.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:19:40.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:19:41.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2026-02-21T01:19:41.218 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:19:41.218 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:19:41.218 INFO:tasks.workunit.client.0.trial106.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 5.4252e-05 s, 22.7 MB/s 2026-02-21T01:19:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2026-02-21T01:19:41.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2026-02-21T01:19:41.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2026-02-21T01:19:41.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-21T01:19:41.273 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2026-02-21T01:19:41.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2026-02-21T01:19:41.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2026-02-21T01:19:41.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2026-02-21T01:19:41.295 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2026-02-21T01:19:41.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2026-02-21T01:19:41.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2026-02-21T01:19:41.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2026-02-21T01:19:41.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2026-02-21T01:19:41.358 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2026-02-21T01:19:41.359 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2026-02-21T01:19:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2026-02-21T01:19:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2026-02-21T01:19:41.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-21T01:19:41.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2026-02-21T01:19:41.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2026-02-21T01:19:41.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2026-02-21T01:19:41.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2026-02-21T01:19:41.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2026-02-21T01:19:41.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-21T01:19:41.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-21T01:19:41.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2026-02-21T01:19:41.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2026-02-21T01:19:41.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2026-02-21T01:19:41.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2026-02-21T01:19:41.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2026-02-21T01:19:41.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2026-02-21T01:19:41.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2026-02-21T01:19:41.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2026-02-21T01:19:41.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2026-02-21T01:19:41.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2026-02-21T01:19:41.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2026-02-21T01:19:41.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2026-02-21T01:19:41.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2026-02-21T01:19:41.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2026-02-21T01:19:41.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-21T01:19:41.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:19:41.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No data available' 2026-02-21T01:19:41.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:19:41.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:19:41.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-21T01:19:41.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-21T01:19:41.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-21T01:19:41.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-21T01:19:41.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No data available' 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:19:41.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-21T01:19:41.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-21T01:19:41.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-21T01:19:41.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-21T01:19:41.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-21T01:19:41.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2026-02-21T01:19:41.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2026-02-21T01:19:41.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-21T01:19:41.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:19:41.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No such file or directory' 2026-02-21T01:19:41.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:19:41.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:19:41.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd --striper stat toyfile 2026-02-21T01:19:41.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-21T01:19:41.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-21T01:19:41.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No such file or directory' 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:19:41.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2026-02-21T01:19:41.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-21T01:19:41.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-21T01:19:41.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-21T01:19:41.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-21T01:19:41.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2026-02-21T01:19:41.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-21T01:19:41.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:41.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-21T01:19:41.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:41.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:41.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:41.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:41.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:41.796 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:41.796 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:41.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:41.797 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:41.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:41.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:41.799 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:41.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:41.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:41.801 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:41.801 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:41.802 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:41.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:41.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-21T01:19:41.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:41.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:41.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:41.812 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55911 2026-02-21T01:19:41.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:41.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:41.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-21T01:19:41.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/rados-striper 0 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:41.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:41.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:41.817 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:41.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:41.818 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:41.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:41.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:41.820 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:41.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:41.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:41.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:41.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:41.823 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:41.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T01:19:41.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-21T01:19:41.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:41.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:41.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55911 2026-02-21T01:19:41.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55911 2026-02-21T01:19:41.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:41.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:41.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-21T01:19:41.828 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:19:41.828 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:19:41.877 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2026-02-21T01:19:41.877 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-21T01:19:41.932 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:19:41.935 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:19:41.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: run_tests: export PATH=./bin:.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:19:41.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: run_tests: PATH=./bin:.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:19:41.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: run_tests: export CEPH_MON=127.0.0.1:7109 2026-02-21T01:19:41.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: run_tests: CEPH_MON=127.0.0.1:7109 2026-02-21T01:19:41.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: run_tests: export CEPH_ARGS 2026-02-21T01:19:41.936 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: run_tests: uuidgen 2026-02-21T01:19:41.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: run_tests: CEPH_ARGS+=' --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none ' 2026-02-21T01:19:41.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:41.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: run_tests: export CEPH_CONF=/dev/null 2026-02-21T01:19:41.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: run_tests: CEPH_CONF=/dev/null 2026-02-21T01:19:41.937 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: run_tests: set 2026-02-21T01:19:41.937 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2026-02-21T01:19:41.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: run_tests: local 'funcs=test_activate_osd 2026-02-21T01:19:41.939 INFO:tasks.workunit.client.0.trial106.stderr:test_activate_osd_after_mark_down 2026-02-21T01:19:41.939 INFO:tasks.workunit.client.0.trial106.stderr:test_activate_osd_skip_benchmark 2026-02-21T01:19:41.939 INFO:tasks.workunit.client.0.trial106.stderr:test_destroy_osd 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_display_logs 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_erasure_code_plugin_exists 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_expect_failure 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_file_with_random_data 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_flush_pg_stats 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_get_config 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_get_is_making_recovery_progress 2026-02-21T01:19:41.940 INFO:tasks.workunit.client.0.trial106.stderr:test_get_last_scrub_stamp 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_not_primary 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_num_active_clean 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_num_active_or_peered 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_num_pgs 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_op_scheduler 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_osd_id_used_by_pgs 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_osds 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_pg 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_primary 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_get_timeout_delays 2026-02-21T01:19:41.941 INFO:tasks.workunit.client.0.trial106.stderr:test_is_clean 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_kill_daemon 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_kill_daemons 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_objectstore_tool 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_pg_schedule_scrub 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_pg_scrub 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_repair 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_run_in_background 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_run_mon 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_run_osd 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_set_config 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_setup 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_teardown 2026-02-21T01:19:41.942 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_background 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_for_clean 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_for_health_ok 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_for_osd 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_for_peered 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_for_scrub 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:test_wait_osd_id_used_by_pgs' 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: run_tests: local dir=td/ceph-helpers 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_activate_osd td/ceph-helpers 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: test_activate_osd: local dir=td/ceph-helpers 2026-02-21T01:19:41.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: test_activate_osd: setup td/ceph-helpers 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:41.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:41.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:41.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:41.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:41.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:41.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:19:41.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:41.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:41.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:41.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:19:41.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:41.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:41.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:19:41.952 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:19:41.952 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:41.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:41.953 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:19:41.954 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:19:41.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:19:41.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:19:41.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:19:41.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: test_activate_osd: run_mon td/ceph-helpers a 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:19:41.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:19:41.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:19:41.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:41.980 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:41.980 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:41.980 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:41.980 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:41.980 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:41.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:19:42.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:19:42.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:19:42.008 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:42.009 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:42.009 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:19:42.009 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:19:42.009 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:42.009 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:42.009 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:42.010 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:42.010 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:42.010 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:42.010 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:19:42.010 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:42.010 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:19:42.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:19:42.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:42.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:42.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:19:42.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:42.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: test_activate_osd: run_mgr td/ceph-helpers x 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:19:42.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:19:42.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:19:42.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:42.238 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:42.238 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:42.238 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:42.238 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:42.238 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:42.238 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:19:42.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:19:42.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:884: test_activate_osd: run_osd td/ceph-helpers 0 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:19:42.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:42.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:19:42.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:19:42.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:19:42.261 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:19:42.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d73ab99d-49ef-42bf-8aa6-0102084b2066 2026-02-21T01:19:42.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d73ab99d-49ef-42bf-8aa6-0102084b2066' 2026-02-21T01:19:42.262 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 d73ab99d-49ef-42bf-8aa6-0102084b2066 2026-02-21T01:19:42.262 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:19:42.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAuCJlpfO4NEBAAhk2IPGw9bc2hdjxrgHoLUw== 2026-02-21T01:19:42.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAuCJlpfO4NEBAAhk2IPGw9bc2hdjxrgHoLUw=="}' 2026-02-21T01:19:42.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d73ab99d-49ef-42bf-8aa6-0102084b2066 -i td/ceph-helpers/0/new.json 2026-02-21T01:19:42.379 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:42.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:19:42.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuCJlpfO4NEBAAhk2IPGw9bc2hdjxrgHoLUw== --osd-uuid d73ab99d-49ef-42bf-8aa6-0102084b2066 2026-02-21T01:19:42.403 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:42.400+0000 7fb014c0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:42.405 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:42.404+0000 7fb014c0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:42.406 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:42.404+0000 7fb014c0da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:42.407 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:42.404+0000 7fb014c0da40 -1 bdev(0x5559a62f8c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:19:42.407 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:42.404+0000 7fb014c0da40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:19:43.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:19:43.109 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:19:43.109 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:19:43.110 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:19:43.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:19:43.234 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:43.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:19:43.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:19:43.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:19:43.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:19:43.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:19:43.259 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:43.256+0000 7f0eb55d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:43.261 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:43.260+0000 7f0eb55d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:43.263 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:43.260+0000 7f0eb55d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:43.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:43.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:43.691 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:43.688+0000 7f0eb55d8a40 -1 Falling back to public interface 2026-02-21T01:19:44.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:44.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:44.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:44.461 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:44.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:44.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:44.679 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:44.675+0000 7f0eb55d8a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:19:45.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:45.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:45.570 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:45.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:19:45.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:45.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:45.789 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/726426967,v1:127.0.0.1:6801/726426967] [v2:127.0.0.1:6802/726426967,v1:127.0.0.1:6803/726426967] exists,up d73ab99d-49ef-42bf-8aa6-0102084b2066 2026-02-21T01:19:45.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: get_asok_path osd.0 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:45.791 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:19:45.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: CEPH_ARGS= 2026-02-21T01:19:45.792 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:19:45.851 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-21T01:19:45.852 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2026-02-21T01:19:45.852 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: grep --quiet osd_max_backfills 2026-02-21T01:19:45.853 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:889: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2026-02-21T01:19:45.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:45.854 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:45.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:45.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:45.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:46.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:46.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-21T01:19:46.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:19:46.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:46.165 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:46.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:19:46.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-21T01:19:46.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:19:46.168 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:46.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:19:46.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-21T01:19:46.170 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T01:19:46.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:19:46.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:19:46.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:19:46.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:19:46.183 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:46.179+0000 7f8ecfcfca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:46.184 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:46.179+0000 7f8ecfcfca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:46.185 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:46.183+0000 7f8ecfcfca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:46.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:19:46.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:46.370 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:46.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:46.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:46.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:46.594 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:46.591+0000 7f8ecfcfca40 -1 Falling back to public interface 2026-02-21T01:19:47.208 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:47.203+0000 7f8ecfcfca40 -1 osd.0 5 log_to_monitors true 2026-02-21T01:19:47.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:47.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:47.588 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:47.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:47.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:47.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:47.804 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/1706415189,v1:127.0.0.1:6801/1706415189] [v2:127.0.0.1:6802/1706415189,v1:127.0.0.1:6803/1706415189] exists,up d73ab99d-49ef-42bf-8aa6-0102084b2066 2026-02-21T01:19:47.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:47.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:47.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:47.805 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: get_asok_path osd.0 2026-02-21T01:19:47.805 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:19:47.805 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:19:47.806 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:47.806 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:47.806 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:47.806 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:19:47.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: CEPH_ARGS= 2026-02-21T01:19:47.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:894: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: teardown td/ceph-helpers 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:47.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:47.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:47.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:47.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:47.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:47.976 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:47.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:47.977 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:47.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:47.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:47.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:47.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:47.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:47.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:47.981 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:47.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:47.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:47.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:19:47.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:47.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:47.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:47.985 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:19:47.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:47.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:47.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:19:47.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2026-02-21T01:19:47.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd_after_mark_down: setup td/ceph-helpers 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:47.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:47.989 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:47.989 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:47.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:47.990 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:47.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:47.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:47.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:47.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:47.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:47.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:47.993 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:47.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:47.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:47.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:19:47.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:47.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:47.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:47.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:19:47.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:47.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:47.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:19:47.999 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:19:47.999 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:47.999 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:47.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:19:48.000 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:904: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:19:48.002 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:48.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:48.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:19:48.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:48.055 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:48.056 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:48.056 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:19:48.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:48.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:48.112 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:48.113 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:48.113 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:48.113 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:48.113 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:19:48.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:48.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:19:48.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2026-02-21T01:19:48.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:19:48.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:19:48.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:19:48.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:19:48.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:19:48.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:19:48.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:19:48.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:48.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:48.288 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:48.288 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:48.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:48.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:48.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:19:48.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:19:48.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:907: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2026-02-21T01:19:48.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:19:48.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:19:48.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:19:48.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:19:48.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:19:48.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:19:48.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:19:48.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:19:48.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 2026-02-21T01:19:48.314 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 2026-02-21T01:19:48.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d4a85dee-8f98-40b2-aee7-e42a7b75bdc9' 2026-02-21T01:19:48.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:19:48.321 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0CJlp+LoTExAAWglFpiK1yZRWjAiF8x48MA== 2026-02-21T01:19:48.321 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0CJlp+LoTExAAWglFpiK1yZRWjAiF8x48MA=="}' 2026-02-21T01:19:48.321 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 -i td/ceph-helpers/0/new.json 2026-02-21T01:19:48.430 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:48.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:19:48.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0CJlp+LoTExAAWglFpiK1yZRWjAiF8x48MA== --osd-uuid d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 2026-02-21T01:19:48.455 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:48.451+0000 7f8ad1c88a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:48.457 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:48.451+0000 7f8ad1c88a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:48.457 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:48.455+0000 7f8ad1c88a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:48.458 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:48.455+0000 7f8ad1c88a40 -1 bdev(0x56115e4e6c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:19:48.458 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:48.455+0000 7f8ad1c88a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:19:49.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:19:49.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:19:49.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:19:49.160 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:19:49.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:19:49.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:19:49.275 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:49.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:19:49.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:19:49.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:19:49.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:19:49.290 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:49.287+0000 7f93f0afaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:49.292 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:49.287+0000 7f93f0afaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:49.293 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:49.291+0000 7f93f0afaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:49.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:19:49.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:49.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:49.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:49.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:49.391 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:49.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:49.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:49.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:49.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:49.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:49.698 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:49.695+0000 7f93f0afaa40 -1 Falling back to public interface 2026-02-21T01:19:50.336 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:50.331+0000 7f93f0afaa40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:19:50.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:50.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:50.502 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:50.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:50.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:50.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:50.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:51.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:51.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:51.640 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:51.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:19:51.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:51.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:51.858 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1396267147,v1:127.0.0.1:6801/1396267147] [v2:127.0.0.1:6802/1396267147,v1:127.0.0.1:6803/1396267147] exists,up d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 2026-02-21T01:19:51.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:51.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:51.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-21T01:19:51.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:19:51.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2026-02-21T01:19:51.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2026-02-21T01:19:51.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2026-02-21T01:19:51.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:912: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2026-02-21T01:19:51.922 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:51.922 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:51.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:51.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:51.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:52.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:52.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: ceph osd down 0 2026-02-21T01:19:52.443 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already down. 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: wait_for_osd down 0 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:52.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T01:19:52.662 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/1396267147,v1:127.0.0.1:6801/1396267147] [v2:127.0.0.1:6802/1396267147,v1:127.0.0.1:6803/1396267147] exists d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:52.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:52.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:19:52.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-21T01:19:52.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:19:52.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:19:52.667 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:52.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-21T01:19:52.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T01:19:52.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:19:52.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:19:52.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:19:52.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:19:52.684 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:52.679+0000 7f94ad06ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:52.686 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:52.683+0000 7f94ad06ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:52.687 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:52.683+0000 7f94ad06ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:52.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:19:52.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:52.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:52.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:52.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:52.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:52.876 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:52.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:52.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:52.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:53.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:53.102 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:53.099+0000 7f94ad06ca40 -1 Falling back to public interface 2026-02-21T01:19:53.797 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:53.791+0000 7f94ad06ca40 -1 osd.0 5 log_to_monitors true 2026-02-21T01:19:54.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:54.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:54.094 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:54.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:54.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:54.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:54.311 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/310804594,v1:127.0.0.1:6801/310804594] [v2:127.0.0.1:6802/310804594,v1:127.0.0.1:6803/310804594] exists,up d4a85dee-8f98-40b2-aee7-e42a7b75bdc9 2026-02-21T01:19:54.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:54.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-21T01:19:54.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:19:54.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:54.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:54.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:54.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:54.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:54.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:54.480 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:54.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:54.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:54.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:54.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:54.483 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:54.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:54.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:54.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:54.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:54.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:54.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:54.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:19:54.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:54.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:54.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:54.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:54.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:19:54.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:19:54.498 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:19:54.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:19:54.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:19:54.499 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:19:54.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:54.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:19:54.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:19:54.501 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:19:54.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:19:54.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:19:54.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:19:54.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:19:54.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:19:54.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:19:54.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:19:54.506 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:19:54.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:19:54.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.507 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:19:54.508 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:19:54.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:19:54.532 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:54.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:19:54.563 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:54.565 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:54.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:19:54.620 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:19:54.621 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:19:54.621 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:19:54.621 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:19:54.621 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:19:54.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:19:54.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:19:54.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:19:54.622 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:54.623 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.623 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.623 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:19:54.623 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:19:54.623 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:19:54.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2026-02-21T01:19:54.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:19:54.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:19:54.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:19:54.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:19:54.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:19:54.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:19:54.798 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:19:54.798 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:54.799 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:54.799 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:54.799 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.799 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.799 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:54.799 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:19:54.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:19:54.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T01:19:54.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:19:54.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:19:54.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:19:54.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:19:54.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:19:54.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:54.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:54.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:19:54.823 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-21T01:19:54.823 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:19:54.823 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:19:54.823 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03b0a39c-990d-4de5-880c-afe52ea10393 2026-02-21T01:19:54.824 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 03b0a39c-990d-4de5-880c-afe52ea10393 2026-02-21T01:19:54.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 03b0a39c-990d-4de5-880c-afe52ea10393' 2026-02-21T01:19:54.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:19:54.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6CJlpnsWcMRAAEc9RS2Spb4jGD/1yIp+t7Q== 2026-02-21T01:19:54.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6CJlpnsWcMRAAEc9RS2Spb4jGD/1yIp+t7Q=="}' 2026-02-21T01:19:54.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03b0a39c-990d-4de5-880c-afe52ea10393 -i td/ceph-helpers/0/new.json 2026-02-21T01:19:54.944 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:54.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:19:54.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQA6CJlpnsWcMRAAEc9RS2Spb4jGD/1yIp+t7Q== --osd-uuid 03b0a39c-990d-4de5-880c-afe52ea10393 2026-02-21T01:19:54.971 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:54.967+0000 7fbbb8052a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:54.972 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:54.967+0000 7fbbb8052a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:54.973 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:54.971+0000 7fbbb8052a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:54.974 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:54.971+0000 7fbbb8052a40 -1 bdev(0x55f68be8ec00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:19:54.974 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:54.971+0000 7fbbb8052a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:19:55.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:19:55.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:19:55.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:19:55.622 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:19:55.622 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:19:55.738 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:55.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:19:55.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T01:19:55.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:19:55.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:19:55.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:19:55.761 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:55.759+0000 7f06a9344a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:55.763 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:55.759+0000 7f06a9344a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:55.764 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:55.759+0000 7f06a9344a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:55.848 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:19:55.848 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:55.848 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:55.848 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:55.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:55.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:55.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:55.849 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:55.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:55.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:55.959 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:56.215 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:56.211+0000 7f06a9344a40 -1 Falling back to public interface 2026-02-21T01:19:56.891 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:56.887+0000 7f06a9344a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:19:56.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:56.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:56.961 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:19:56.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:19:56.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:56.965 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:57.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:19:58.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:19:58.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:58.094 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:19:58.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:19:58.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:58.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:58.311 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1746319065,v1:127.0.0.1:6801/1746319065] [v2:127.0.0.1:6802/1746319065,v1:127.0.0.1:6803/1746319065] exists,up 03b0a39c-990d-4de5-880c-afe52ea10393 2026-02-21T01:19:58.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:58.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:58.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:58.312 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:19:58.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T01:19:58.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-21T01:19:58.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-21T01:19:58.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T01:19:58.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:19:58.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:19:58.373 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:19:58.374 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:58.374 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:58.374 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:19:58.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T01:19:58.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-21T01:19:58.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-21T01:19:58.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:940: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2026-02-21T01:19:58.435 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:19:58.435 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:19:58.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:19:58.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:19:58.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:19:58.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:19:58.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:941: test_activate_osd_skip_benchmark: ceph osd down 0 2026-02-21T01:19:58.948 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already down. 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: wait_for_osd down 0 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:58.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:58.959 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:58.959 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:58.959 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T01:19:59.172 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/1746319065,v1:127.0.0.1:6801/1746319065] [v2:127.0.0.1:6802/1746319065,v1:127.0.0.1:6803/1746319065] exists 03b0a39c-990d-4de5-880c-afe52ea10393 2026-02-21T01:19:59.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:19:59.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:19:59.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:946: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:19:59.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:19:59.177 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:19:59.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:19:59.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:19:59.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:19:59.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-21T01:19:59.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:19:59.179 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:19:59.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:19:59.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T01:19:59.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T01:19:59.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:19:59.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:19:59.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:19:59.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:19:59.194 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:59.191+0000 7f45ee0bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:59.196 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:59.191+0000 7f45ee0bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:59.196 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:59.191+0000 7f45ee0bfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:19:59.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:19:59.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:19:59.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:19:59.539 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:19:59.535+0000 7f45ee0bfa40 -1 Falling back to public interface 2026-02-21T01:19:59.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:00.209 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:00.203+0000 7f45ee0bfa40 -1 osd.0 6 log_to_monitors true 2026-02-21T01:20:00.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:00.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:00.617 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:00.617 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:00.617 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:00.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:00.827 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/2247285599,v1:127.0.0.1:6801/2247285599] [v2:127.0.0.1:6802/2247285599,v1:127.0.0.1:6803/2247285599] exists,up 03b0a39c-990d-4de5-880c-afe52ea10393 2026-02-21T01:20:00.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:00.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:00.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:00.829 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T01:20:00.829 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:20:00.829 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:20:00.830 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:00.830 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:00.830 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:00.830 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:20:00.830 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T01:20:00.830 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-21T01:20:00.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-21T01:20:00.887 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T01:20:00.887 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:20:00.887 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:20:00.888 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:00.888 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:00.888 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:00.888 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:20:00.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T01:20:00.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-21T01:20:00.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-21T01:20:00.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:953: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-21T01:20:00.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:954: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:00.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:01.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:01.058 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:01.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:01.059 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:01.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:01.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:01.061 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:01.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:01.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:01.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:01.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:01.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:01.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:01.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:01.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:01.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_destroy_osd td/ceph-helpers 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: test_destroy_osd: local dir=td/ceph-helpers 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: test_destroy_osd: setup td/ceph-helpers 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:01.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:01.073 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:01.074 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:01.074 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:01.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:01.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:01.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:01.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:01.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:01.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:01.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:01.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:01.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:01.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:01.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:01.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:01.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:01.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:01.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:01.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:01.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: test_destroy_osd: run_mon td/ceph-helpers a 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:01.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:01.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:01.109 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:20:01.141 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:01.141 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:01.141 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:01.141 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:01.141 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.142 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.143 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:01.143 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:01.143 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:01.200 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:01.201 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:01.202 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:01.202 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.203 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.203 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:01.203 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:01.203 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: test_destroy_osd: run_mgr td/ceph-helpers x 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:01.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:01.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:01.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:01.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:01.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:01.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:01.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:01.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: test_destroy_osd: run_osd td/ceph-helpers 0 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:01.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:01.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:01.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:01.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=064b6d9b-2d9f-48b6-8a31-795df5f3dfcf 2026-02-21T01:20:01.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 064b6d9b-2d9f-48b6-8a31-795df5f3dfcf' 2026-02-21T01:20:01.395 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 064b6d9b-2d9f-48b6-8a31-795df5f3dfcf 2026-02-21T01:20:01.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:01.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBCJlpXlwtGBAAF4dNtT2zK9ghycXftL31qw== 2026-02-21T01:20:01.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBCJlpXlwtGBAAF4dNtT2zK9ghycXftL31qw=="}' 2026-02-21T01:20:01.406 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 064b6d9b-2d9f-48b6-8a31-795df5f3dfcf -i td/ceph-helpers/0/new.json 2026-02-21T01:20:01.516 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:01.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:01.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBCJlpXlwtGBAAF4dNtT2zK9ghycXftL31qw== --osd-uuid 064b6d9b-2d9f-48b6-8a31-795df5f3dfcf 2026-02-21T01:20:01.541 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:01.539+0000 7f86e6b93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:01.543 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:01.539+0000 7f86e6b93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:01.544 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:01.539+0000 7f86e6b93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:01.544 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:01.539+0000 7f86e6b93a40 -1 bdev(0x55816f8e0c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:01.544 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:01.539+0000 7f86e6b93a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:02.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:02.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:02.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:02.234 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:02.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:02.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:02.346 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:02.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:02.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:02.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:02.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:02.361 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:02.359+0000 7fc090500a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:02.363 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:02.359+0000 7fc090500a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:02.363 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:02.359+0000 7fc090500a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:02.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:02.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:02.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:02.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:02.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:02.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:02.460 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:02.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:02.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:02.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:02.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:02.778 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:02.775+0000 7fc090500a40 -1 Falling back to public interface 2026-02-21T01:20:03.435 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:03.431+0000 7fc090500a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:03.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:03.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:03.568 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:03.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:03.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:03.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:03.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:04.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:04.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:04.734 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:04.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:04.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:04.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:04.951 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/369648905,v1:127.0.0.1:6801/369648905] [v2:127.0.0.1:6802/369648905,v1:127.0.0.1:6803/369648905] exists,up 064b6d9b-2d9f-48b6-8a31-795df5f3dfcf 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:793: test_destroy_osd: destroy_osd td/ceph-helpers 0 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:775: destroy_osd: local dir=td/ceph-helpers 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:776: destroy_osd: local id=0 2026-02-21T01:20:04.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:778: destroy_osd: ceph osd out osd.0 2026-02-21T01:20:05.214 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already out. 2026-02-21T01:20:05.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:779: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T01:20:05.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:05.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:05.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:05.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:05.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:05.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:05.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:780: destroy_osd: ceph osd down osd.0 2026-02-21T01:20:05.739 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already down. 2026-02-21T01:20:05.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:781: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2026-02-21T01:20:05.951 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 does not exist 2026-02-21T01:20:05.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: teardown td/ceph-helpers/0 2026-02-21T01:20:05.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers/0 2026-02-21T01:20:05.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:05.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers/0 KILL 2026-02-21T01:20:05.962 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:05.962 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:05.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:05.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:05.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:05.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:05.964 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:05.965 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:05.965 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:05.966 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:05.966 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:05.967 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:05.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:05.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:05.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:05.969 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:05.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:05.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:05.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers/0 2026-02-21T01:20:05.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:05.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:05.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:05.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:05.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:05.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:05.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: rm -fr td/ceph-helpers/0 2026-02-21T01:20:05.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: ceph osd dump 2026-02-21T01:20:05.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: grep 'osd. ' 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:795: test_destroy_osd: teardown td/ceph-helpers 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:06.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:06.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:06.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:06.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:06.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:06.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:06.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:06.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:06.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:06.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:06.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:06.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:06.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:06.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:06.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:06.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_display_logs td/ceph-helpers 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_display_logs: local dir=td/ceph-helpers 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_display_logs: setup td/ceph-helpers 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:06.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:06.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:06.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:06.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:06.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:06.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:06.321 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:06.321 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:06.322 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:06.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:06.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:06.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:06.324 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:06.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:06.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:06.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:06.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:06.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:06.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:06.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.330 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:06.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:06.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:06.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:06.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:06.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:06.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:06.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:06.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:06.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_display_logs: run_mon td/ceph-helpers a 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:06.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:20:06.356 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:06.356 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:06.356 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:06.357 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:06.357 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.357 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.357 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:06.357 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:20:06.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:06.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:06.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:06.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:06.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:06.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:06.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:06.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:06.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:06.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:06.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:06.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:06.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:06.460 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:06.461 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:06.461 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:06.462 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:06.462 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.462 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.462 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:06.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:06.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:06.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_display_logs: kill_daemons td/ceph-helpers 2026-02-21T01:20:06.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:06.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:06.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:06.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:06.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:06.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:06.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_display_logs: display_logs td/ceph-helpers 2026-02-21T01:20:06.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: display_logs: local dir=td/ceph-helpers 2026-02-21T01:20:06.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2026-02-21T01:20:06.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: display_logs: read file 2026-02-21T01:20:06.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2026-02-21T01:20:06.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: display_logs: cat td/ceph-helpers/mon.a.log 2026-02-21T01:20:06.626 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: display_logs: read file 2026-02-21T01:20:06.627 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2164: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_display_logs: teardown td/ceph-helpers 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:06.629 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:06.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:06.631 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:06.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:06.632 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:06.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:06.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:06.633 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:06.634 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:06.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:06.635 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:06.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:06.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:06.639 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:06.639 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.640 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:06.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:06.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:06.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:06.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: test_erasure_code_plugin_exists: setup td/ceph-helpers 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:06.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:06.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:06.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:06.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:06.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:06.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:06.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:06.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:06.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:06.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:06.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:06.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:06.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:06.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:06.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:06.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:06.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:06.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:06.654 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:06.654 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.654 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:06.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:06.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:06.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:06.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:06.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:20:06.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:06.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:06.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:06.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:06.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:06.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:06.706 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:06.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:06.763 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:06.763 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:06.763 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:06.763 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.764 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:06.765 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:06.765 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:06.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2026-02-21T01:20:06.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:06.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:06.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:06.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:06.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:06.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:06.925 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:06.925 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:06.925 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:06.926 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:06.926 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:06.926 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:06.926 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:06.926 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:06.927 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:06.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2026-02-21T01:20:06.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=jerasure 2026-02-21T01:20:06.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-21T01:20:06.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-21T01:20:06.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-21T01:20:06.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-21T01:20:06.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-21T01:20:06.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2026-02-21T01:20:06.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2026-02-21T01:20:07.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s= 2026-02-21T01:20:07.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=0 2026-02-21T01:20:07.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-21T01:20:07.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-21T01:20:07.224 INFO:tasks.workunit.client.0.trial106.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 0 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=FAKE 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-21T01:20:07.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-21T01:20:07.233 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-21T01:20:07.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2026-02-21T01:20:07.233 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2026-02-21T01:20:07.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2026-02-21T01:20:07.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=5 2026-02-21T01:20:07.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2026-02-21T01:20:07.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2026-02-21T01:20:07.329 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2026-02-21T01:20:07.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2119: erasure_code_plugin_exists: status=1 2026-02-21T01:20:07.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 1 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:07.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:07.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:07.438 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:07.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:07.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:07.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:07.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:07.440 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:07.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:07.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:07.442 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.443 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:07.444 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:07.444 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:07.446 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:07.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_expect_failure td/ceph-helpers 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_expect_failure: local dir=td/ceph-helpers 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_expect_failure: setup td/ceph-helpers 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:07.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:07.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:07.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:07.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:07.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:07.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:07.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:07.454 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:07.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:07.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:07.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:07.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:07.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:07.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:07.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:07.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:07.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:07.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:07.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:07.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.462 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:07.464 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:20:07.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:20:07.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo FAIL ; exit 1' 2026-02-21T01:20:07.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-21T01:20:07.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-21T01:20:07.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T01:20:07.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-21T01:20:07.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2042: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2026-02-21T01:20:07.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-21T01:20:07.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:20:07.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-21T01:20:07.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:20:07.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:20:07.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo FAIL ; exit 0' 2026-02-21T01:20:07.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: expect_failure: success=true 2026-02-21T01:20:07.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: true 2026-02-21T01:20:07.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: cat td/ceph-helpers/out 2026-02-21T01:20:07.469 INFO:tasks.workunit.client.0.trial106.stderr:FAIL 2026-02-21T01:20:07.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: expect_failure: return 1 2026-02-21T01:20:07.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2043: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-21T01:20:07.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2026-02-21T01:20:07.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-21T01:20:07.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-21T01:20:07.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T01:20:07.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: cat td/ceph-helpers/out 2026-02-21T01:20:07.474 INFO:tasks.workunit.client.0.trial106.stderr:UNEXPECTED 2026-02-21T01:20:07.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: expect_failure: return 1 2026-02-21T01:20:07.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T01:20:07.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_expect_failure: teardown td/ceph-helpers 2026-02-21T01:20:07.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:07.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:07.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:07.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:07.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:07.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:07.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:07.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:07.479 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:07.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:07.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:07.480 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:07.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:07.483 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:07.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:07.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:07.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:07.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:07.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_file_with_random_data td/ceph-helpers 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: test_file_with_random_data: local dir=td/ceph-helpers 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: test_file_with_random_data: setup td/ceph-helpers 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:07.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:07.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:07.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:07.491 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:07.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:07.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:07.493 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:07.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:07.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:07.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:07.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:07.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:07.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:07.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:07.500 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:07.500 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.500 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:07.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:07.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:07.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:07.502 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:07.502 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.502 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:07.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:07.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:07.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:07.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:07.505 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: test_file_with_random_data: file_with_random_data 4000 2026-02-21T01:20:07.505 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=4000 2026-02-21T01:20:07.505 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-21T01:20:07.506 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.1ri1cG41Pl 2026-02-21T01:20:07.506 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.1ri1cG41Pl bs=4000 count=1 2026-02-21T01:20:07.507 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:20:07.507 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:20:07.507 INFO:tasks.workunit.client.0.trial106.stderr:4000 bytes (4.0 kB, 3.9 KiB) copied, 6.2516e-05 s, 64.0 MB/s 2026-02-21T01:20:07.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.1ri1cG41Pl 2026-02-21T01:20:07.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: test_file_with_random_data: local file=/tmp/tmp.1ri1cG41Pl 2026-02-21T01:20:07.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_file_with_random_data: test -f /tmp/tmp.1ri1cG41Pl 2026-02-21T01:20:07.508 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: test_file_with_random_data: stat -c %s /tmp/tmp.1ri1cG41Pl 2026-02-21T01:20:07.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: test_file_with_random_data: test 4000 = 4000 2026-02-21T01:20:07.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: test_file_with_random_data: rm /tmp/tmp.1ri1cG41Pl 2026-02-21T01:20:07.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2358: test_file_with_random_data: teardown td/ceph-helpers 2026-02-21T01:20:07.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:07.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:07.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:07.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:07.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:07.511 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:07.511 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:07.511 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:07.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:07.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:07.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:07.514 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:07.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:07.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:07.516 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:07.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:07.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:07.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:07.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:07.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:07.521 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:07.521 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.521 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:07.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:07.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:07.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:07.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_flush_pg_stats td/ceph-helpers 2026-02-21T01:20:07.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: test_flush_pg_stats: local dir=td/ceph-helpers 2026-02-21T01:20:07.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: test_flush_pg_stats: setup td/ceph-helpers 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:07.524 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:07.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:07.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:07.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:07.526 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:07.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:07.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:07.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:07.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:07.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:07.529 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:07.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:07.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:07.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:07.532 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:07.532 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.532 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:07.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:07.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:07.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:07.535 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:07.535 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.535 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.535 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:07.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:07.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:07.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:07.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:07.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:07.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:07.560 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:07.560 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.560 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.561 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:07.561 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:07.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:07.591 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:07.591 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:07.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:07.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:07.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:07.593 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:07.593 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:07.593 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:07.594 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:07.594 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.594 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.594 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:07.594 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:07.594 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:07.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:07.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:07.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:07.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:07.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:07.651 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:07.651 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:07.651 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:07.651 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:07.652 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.652 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.652 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:07.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:07.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: test_flush_pg_stats: run_mgr td/ceph-helpers x 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:07.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:07.823 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:07.823 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:07.823 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:07.823 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.823 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.823 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:07.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:07.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: test_flush_pg_stats: run_osd td/ceph-helpers 0 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:07.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:07.843 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:07.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:07.845 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e92ef6b3-e376-42ee-b8bb-5d66c72113b4 2026-02-21T01:20:07.845 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 e92ef6b3-e376-42ee-b8bb-5d66c72113b4 2026-02-21T01:20:07.845 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e92ef6b3-e376-42ee-b8bb-5d66c72113b4' 2026-02-21T01:20:07.845 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:07.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHCJlp5dYVMxAAzgn4fK7N21e9EmcgbiVvoQ== 2026-02-21T01:20:07.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHCJlp5dYVMxAAzgn4fK7N21e9EmcgbiVvoQ=="}' 2026-02-21T01:20:07.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e92ef6b3-e376-42ee-b8bb-5d66c72113b4 -i td/ceph-helpers/0/new.json 2026-02-21T01:20:07.967 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:07.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:07.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHCJlp5dYVMxAAzgn4fK7N21e9EmcgbiVvoQ== --osd-uuid e92ef6b3-e376-42ee-b8bb-5d66c72113b4 2026-02-21T01:20:07.992 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:07.987+0000 7f3f4a466a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:07.993 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:07.991+0000 7f3f4a466a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:07.994 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:07.991+0000 7f3f4a466a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:07.995 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:07.991+0000 7f3f4a466a40 -1 bdev(0x55a2435d6c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:07.995 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:07.991+0000 7f3f4a466a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:08.780 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:08.780 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:08.781 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:08.781 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:08.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:08.906 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:08.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:08.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:08.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:08.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:08.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:08.929 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:08.923+0000 7ffb61a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:08.931 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:08.927+0000 7ffb61a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:08.932 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:08.927+0000 7ffb61a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:09.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:09.126 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:09.434 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:09.431+0000 7ffb61a92a40 -1 Falling back to public interface 2026-02-21T01:20:10.127 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:10.127 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:10.128 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:10.128 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:10.128 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:10.128 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:10.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:10.427 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:10.423+0000 7ffb61a92a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:11.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:11.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:11.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:11.341 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:11.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:11.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:11.556 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2022950108,v1:127.0.0.1:6801/2022950108] [v2:127.0.0.1:6802/2022950108,v1:127.0.0.1:6803/2022950108] exists,up e92ef6b3-e376-42ee-b8bb-5d66c72113b4 2026-02-21T01:20:11.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:11.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:11.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:11.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_flush_pg_stats: create_rbd_pool 2026-02-21T01:20:11.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:20:11.767 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:20:11.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:20:11.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:20:12.054 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:20:12.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:20:13.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:20:13.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_flush_pg_stats: rados -p rbd put obj /etc/group 2026-02-21T01:20:13.367 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: test_flush_pg_stats: flush_pg_stats 2026-02-21T01:20:13.367 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:20:13.367 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:20:13.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:20:13.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:20:13.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:20:13.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:20:13.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:20:13.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:20:13.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:20:13.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:20:13.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:13.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:20:13.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:20:13.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:13.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:20:13.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:20:13.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:20:13.672 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:20:13.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:13.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:20:13.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:20:14.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:20:14.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:15.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:20:15.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2026-02-21T01:20:15.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: ceph df detail --format=json 2026-02-21T01:20:15.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2026-02-21T01:20:15.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: stored=8398 2026-02-21T01:20:15.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: test_flush_pg_stats: ceph df detail --format=json 2026-02-21T01:20:15.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: test_flush_pg_stats: stored_raw=8398 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2293: test_flush_pg_stats: test 8398 -gt 0 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: test_flush_pg_stats: test 8398 == 8398 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: test_flush_pg_stats: teardown td/ceph-helpers 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:15.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:15.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:15.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:15.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:15.763 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:15.764 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:15.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:15.765 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:15.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:15.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:15.767 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:15.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:15.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:15.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:15.769 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:15.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:15.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:15.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:15.776 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:15.776 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.776 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.776 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_config td/ceph-helpers 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: test_get_config: local dir=td/ceph-helpers 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: test_get_config: setup td/ceph-helpers 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:15.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:15.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:15.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:15.779 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:15.780 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:15.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:15.781 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:15.782 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:15.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:15.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:15.783 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:15.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:15.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:15.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:15.785 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:15.786 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:15.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:15.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:15.788 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:15.788 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.788 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:15.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:15.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:15.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:15.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:15.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:15.793 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:15.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:15.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:15.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1131: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:15.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:15.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:15.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:15.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:15.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:15.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:15.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:15.845 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.846 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:15.847 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:15.847 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:15.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:15.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:15.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:15.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.901 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:15.902 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:15.902 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: get_config mon a osd_pool_default_size 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_pool_default_size 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_pool_default_size 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:15.956 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:15.957 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:15.958 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:15.958 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:15.958 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:15.958 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:15.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:15.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get osd_pool_default_size 2026-02-21T01:20:16.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: test 1 = 1 2026-02-21T01:20:16.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: test_get_config: run_mgr td/ceph-helpers x 2026-02-21T01:20:16.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:16.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:16.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:16.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:16.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:16.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:16.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:16.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:16.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:16.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:16.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:16.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:16.130 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:16.130 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:16.154 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:16.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:16.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:16.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_max_scrubs=3 2026-02-21T01:20:16.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:16.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:16.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8b773aaa-5f92-4fdf-a74e-59a15d3dfdc6 2026-02-21T01:20:16.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8b773aaa-5f92-4fdf-a74e-59a15d3dfdc6' 2026-02-21T01:20:16.157 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 8b773aaa-5f92-4fdf-a74e-59a15d3dfdc6 2026-02-21T01:20:16.157 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:16.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQCJlp/1IVChAACUljCTkShwyZbJop4eaqZQ== 2026-02-21T01:20:16.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQCJlp/1IVChAACUljCTkShwyZbJop4eaqZQ=="}' 2026-02-21T01:20:16.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8b773aaa-5f92-4fdf-a74e-59a15d3dfdc6 -i td/ceph-helpers/0/new.json 2026-02-21T01:20:16.286 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:16.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:16.295 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQBQCJlp/1IVChAACUljCTkShwyZbJop4eaqZQ== --osd-uuid 8b773aaa-5f92-4fdf-a74e-59a15d3dfdc6 2026-02-21T01:20:16.311 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:16.306+0000 7f3343354a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:16.312 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:16.306+0000 7f3343354a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:16.314 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:16.310+0000 7f3343354a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:16.314 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:16.310+0000 7f3343354a40 -1 bdev(0x555a6bd96c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:16.315 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:16.310+0000 7f3343354a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:17.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:17.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:17.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:17.049 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:17.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:17.168 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:17.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:17.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2026-02-21T01:20:17.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:17.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:17.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:17.195 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:17.190+0000 7f7bb2210a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:17.197 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:17.194+0000 7f7bb2210a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:17.198 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:17.194+0000 7f7bb2210a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:17.276 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:17.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:17.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:17.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:17.614 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:17.610+0000 7f7bb2210a40 -1 Falling back to public interface 2026-02-21T01:20:18.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:18.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:18.384 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:18.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:18.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:18.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:18.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:18.603 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:18.598+0000 7f7bb2210a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:19.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:19.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:19.492 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:19.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:19.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:19.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:19.702 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/552670209,v1:127.0.0.1:6801/552670209] [v2:127.0.0.1:6802/552670209,v1:127.0.0.1:6803/552670209] exists,up 8b773aaa-5f92-4fdf-a74e-59a15d3dfdc6 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: get_config osd 0 osd_max_scrubs 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_max_scrubs 2026-02-21T01:20:19.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_max_scrubs 2026-02-21T01:20:19.705 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-21T01:20:19.705 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:20:19.705 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:20:19.706 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:19.706 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:19.706 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:19.706 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:20:19.706 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:19.706 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_max_scrubs 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: test 3 = 3 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: teardown td/ceph-helpers 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:19.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:20.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:20.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:20.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:20.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:20.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:20.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:20.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:20.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:20.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:20.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:20.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:20.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:20.088 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:20.089 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.089 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.089 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:20.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:20.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: test_get_is_making_recovery_progress: setup td/ceph-helpers 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:20.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:20.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:20.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:20.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:20.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:20.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:20.094 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:20.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:20.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:20.096 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:20.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:20.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:20.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:20.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:20.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:20.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:20.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:20.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:20.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:20.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:20.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:20.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:20.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:20.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:20.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:20:20.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:20.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:20.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:20.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:20.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.130 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:20.130 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:20:20.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:20.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:20.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:20.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:20.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:20.163 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:20.163 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:20.164 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:20.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:20.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:20.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.220 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.221 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:20.221 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:20.221 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:20.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2026-02-21T01:20:20.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:20.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:20.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:20.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:20.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:20.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:20.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:20.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:20.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:20.391 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:20.391 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.391 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:20.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:20.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:20.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2026-02-21T01:20:20.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:20:20.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:20:20.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:20:20.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:20:20.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:20:20.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:20:20.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:20:20.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1341: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:20.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:20.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:20.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:20.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:20.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:20.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:20.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:20.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:20.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:20.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:20.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:20.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:20.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:20.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:20.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:20.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1524: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: test_get_last_scrub_stamp: setup td/ceph-helpers 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:20.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:20.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:20.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:20.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:20.659 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:20.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:20.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:20.660 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:20.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:20.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:20.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:20.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:20.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:20.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:20.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:20.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:20.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:20.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:20.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:20.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:20.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:20.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:20.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:20.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.692 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:20.693 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:20.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:20.720 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.721 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.722 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:20.722 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:20.722 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:20.777 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:20.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:20.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:20.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:20.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:20.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:20.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:20.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:20.779 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:20.780 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.780 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.780 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:20.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:20.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:20.834 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2026-02-21T01:20:20.834 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:20.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:20.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:20.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:20.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:20.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:20.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:20.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:20.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:20.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:20.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.949 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:20.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:20.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:20.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2026-02-21T01:20:20.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:20.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:20.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:20.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:20.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:20.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:20.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:20.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:20.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:20.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:20.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:20.975 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:20.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:20.978 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d319c3ca-5cb7-4170-8619-022cd164df6c 2026-02-21T01:20:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d319c3ca-5cb7-4170-8619-022cd164df6c' 2026-02-21T01:20:20.979 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 d319c3ca-5cb7-4170-8619-022cd164df6c 2026-02-21T01:20:20.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:20.983 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBUCJlp3ECIOhAA/QkctNfsn2ZxIxJatr+pwA== 2026-02-21T01:20:20.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBUCJlp3ECIOhAA/QkctNfsn2ZxIxJatr+pwA=="}' 2026-02-21T01:20:20.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d319c3ca-5cb7-4170-8619-022cd164df6c -i td/ceph-helpers/0/new.json 2026-02-21T01:20:21.092 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:21.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:21.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBUCJlp3ECIOhAA/QkctNfsn2ZxIxJatr+pwA== --osd-uuid d319c3ca-5cb7-4170-8619-022cd164df6c 2026-02-21T01:20:21.118 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:21.114+0000 7fdb269b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:21.120 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:21.114+0000 7fdb269b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:21.120 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:21.114+0000 7fdb269b1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:21.121 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:21.118+0000 7fdb269b1a40 -1 bdev(0x56272caecc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:21.121 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:21.118+0000 7fdb269b1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:21.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:21.888 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:21.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:21.889 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:21.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:22.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:22.016 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:22.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:22.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:22.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:22.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:22.039 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:22.034+0000 7f29c5daba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:22.042 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:22.038+0000 7f29c5daba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:22.043 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:22.038+0000 7f29c5daba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:22.129 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:22.130 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:22.130 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:22.530 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:22.526+0000 7f29c5daba40 -1 Falling back to public interface 2026-02-21T01:20:23.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:23.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:23.245 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:23.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:23.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:23.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:23.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:23.475 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:23.470+0000 7f29c5daba40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:24.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:24.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:24.463 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:24.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:24.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:24.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:24.678 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/149836963,v1:127.0.0.1:6801/149836963] [v2:127.0.0.1:6802/149836963,v1:127.0.0.1:6803/149836963] exists,up d319c3ca-5cb7-4170-8619-022cd164df6c 2026-02-21T01:20:24.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:24.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:24.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:24.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: test_get_last_scrub_stamp: create_rbd_pool 2026-02-21T01:20:24.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:20:24.879 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:20:24.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:20:24.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:20:25.133 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:20:25.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:20:26.146 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:20:26.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: test_get_last_scrub_stamp: wait_for_clean 2026-02-21T01:20:26.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:20:26.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:20:26.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:20:26.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:20:26.427 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:20:26.427 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:20:26.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:20:26.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:20:26.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:20:26.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:20:26.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:20:26.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:20:26.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:20:26.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:20:26.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:20:26.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:20:26.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:20:26.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:20:26.744 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:20:26.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:20:26.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:20:26.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:20:26.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:20:26.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:26.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:20:26.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:20:26.826 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:26.826 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:20:26.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:20:26.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:20:26.828 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:20:26.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:27.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:20:27.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:20:28.037 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:20:28.037 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:28.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:20:28.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:20:28.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:28.254 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:28.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:20:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:20:28.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:20:28.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:20:28.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:28.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:29.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:20:29.037 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:20:29.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: stamp=2026-02-21T01:20:25.081542+0000 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: test -n 2026-02-21T01:20:25.081542+0000 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1534: test_get_last_scrub_stamp: teardown td/ceph-helpers 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:29.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:29.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:29.558 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:29.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:29.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:29.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:29.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:29.562 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:29.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:29.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:29.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:29.564 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:29.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:29.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:29.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:29.572 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:29.572 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.572 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:29.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:29.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:29.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_not_primary td/ceph-helpers 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: test_get_not_primary: local dir=td/ceph-helpers 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: test_get_not_primary: setup td/ceph-helpers 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:29.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:29.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:29.576 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:29.576 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:29.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:29.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:29.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:29.577 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:29.577 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:29.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:29.578 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:29.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:29.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:29.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:29.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:29.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:29.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:29.581 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:29.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:29.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:29.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:29.583 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:29.583 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.584 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:29.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:29.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:29.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:29.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:29.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:29.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-21T01:20:29.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:29.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:29.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:29.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:29.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.609 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.610 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:29.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-21T01:20:29.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:29.641 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:29.641 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:29.641 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:29.641 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.642 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.643 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:29.643 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:29.643 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:29.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:29.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:29.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:29.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:29.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:29.697 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:29.698 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:29.698 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:29.698 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:29.698 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.698 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.698 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:29.699 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:29.699 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:29.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: test_get_not_primary: run_mgr td/ceph-helpers x 2026-02-21T01:20:29.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:29.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:29.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:29.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:29.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:29.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:29.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:29.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:29.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:29.871 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:29.871 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.871 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.871 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:29.871 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:29.872 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: test_get_not_primary: run_osd td/ceph-helpers 0 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:29.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:29.892 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:29.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:29.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:29.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:29.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5d1d4daf-e2f4-44df-99fa-07c2537ef745 2026-02-21T01:20:29.895 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 5d1d4daf-e2f4-44df-99fa-07c2537ef745 2026-02-21T01:20:29.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5d1d4daf-e2f4-44df-99fa-07c2537ef745' 2026-02-21T01:20:29.895 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:29.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBdCJlpzCjqNRAAePglFjROs7e3j3upk/L4hA== 2026-02-21T01:20:29.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBdCJlpzCjqNRAAePglFjROs7e3j3upk/L4hA=="}' 2026-02-21T01:20:29.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5d1d4daf-e2f4-44df-99fa-07c2537ef745 -i td/ceph-helpers/0/new.json 2026-02-21T01:20:30.018 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:30.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:30.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdCJlpzCjqNRAAePglFjROs7e3j3upk/L4hA== --osd-uuid 5d1d4daf-e2f4-44df-99fa-07c2537ef745 2026-02-21T01:20:30.044 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.038+0000 7f92294d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:30.045 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.042+0000 7f92294d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:30.046 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.042+0000 7f92294d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:30.046 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.042+0000 7f92294d1a40 -1 bdev(0x558a8302ec00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:30.046 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.042+0000 7f92294d1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:30.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:30.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:30.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:30.790 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:30.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:30.913 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:30.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:30.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:30.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:30.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:30.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:30.937 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.930+0000 7f16b27c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:30.939 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.934+0000 7f16b27c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:30.940 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:30.934+0000 7f16b27c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:31.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:31.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:31.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:31.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:31.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:31.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:31.022 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:31.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:31.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:31.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:31.135 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:31.434 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:31.430+0000 7f16b27c5a40 -1 Falling back to public interface 2026-02-21T01:20:32.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:32.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:32.137 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:32.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:32.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:32.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:32.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:32.411 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:32.406+0000 7f16b27c5a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:33.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:33.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:33.246 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:33.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:33.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:33.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:33.451 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1470748051,v1:127.0.0.1:6801/1470748051] [v2:127.0.0.1:6802/1470748051,v1:127.0.0.1:6803/1470748051] exists,up 5d1d4daf-e2f4-44df-99fa-07c2537ef745 2026-02-21T01:20:33.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:33.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:33.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: test_get_not_primary: run_osd td/ceph-helpers 1 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:20:33.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:33.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:33.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:20:33.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:33.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ee6825d7-f8ac-417a-9653-039fd6b40cdb 2026-02-21T01:20:33.459 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 ee6825d7-f8ac-417a-9653-039fd6b40cdb 2026-02-21T01:20:33.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ee6825d7-f8ac-417a-9653-039fd6b40cdb' 2026-02-21T01:20:33.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:33.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBhCJlptS/dGxAATkTcxvPlMfFMtL081873Lw== 2026-02-21T01:20:33.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBhCJlptS/dGxAATkTcxvPlMfFMtL081873Lw=="}' 2026-02-21T01:20:33.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ee6825d7-f8ac-417a-9653-039fd6b40cdb -i td/ceph-helpers/1/new.json 2026-02-21T01:20:33.682 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:33.691 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:20:33.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhCJlptS/dGxAATkTcxvPlMfFMtL081873Lw== --osd-uuid ee6825d7-f8ac-417a-9653-039fd6b40cdb 2026-02-21T01:20:33.709 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:33.706+0000 7f41f4f38a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:33.710 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:33.706+0000 7f41f4f38a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:33.711 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:33.706+0000 7f41f4f38a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:33.712 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:33.706+0000 7f41f4f38a40 -1 bdev(0x5653654d8c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:33.712 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:33.706+0000 7f41f4f38a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:20:34.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:20:34.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:34.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:20:34.385 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:20:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:34.673 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:20:34.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:20:34.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:34.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:34.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:34.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:34.697 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:34.690+0000 7fd504aaca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:34.698 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:34.694+0000 7fd504aaca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:34.699 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:34.694+0000 7fd504aaca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:34.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:20:34.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:34.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:20:34.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:34.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:34.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:34.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:34.894 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:34.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:34.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:20:35.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:35.162 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:35.158+0000 7fd504aaca40 -1 Falling back to public interface 2026-02-21T01:20:35.711 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:35.706+0000 7fd504aaca40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:20:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:36.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:36.099 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:36.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:36.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:36.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:20:36.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:37.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:37.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:37.311 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:37.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:37.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:37.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:20:37.517 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1767930462,v1:127.0.0.1:6811/1767930462] [v2:127.0.0.1:6812/1767930462,v1:127.0.0.1:6813/1767930462] exists,up ee6825d7-f8ac-417a-9653-039fd6b40cdb 2026-02-21T01:20:37.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:37.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:37.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:37.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: test_get_not_primary: create_rbd_pool 2026-02-21T01:20:37.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:20:37.719 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:20:37.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:20:37.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:20:37.971 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:20:37.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:20:38.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: test_get_not_primary: wait_for_clean 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:20:39.255 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:20:39.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:20:39.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:20:39.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:20:39.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:20:39.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:20:39.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:20:39.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:20:39.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:20:39.350 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:20:39.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:20:39.569 INFO:tasks.workunit.client.0.trial106.stderr:1' 2026-02-21T01:20:39.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:20:39.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:20:39.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:20:39.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-21T01:20:39.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-21T01:20:39.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T01:20:39.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:20:39.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:20:39.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-21T01:20:39.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-21T01:20:39.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2026-02-21T01:20:39.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:20:39.730 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-21T01:20:39.730 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:20:39.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:20:39.731 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-21T01:20:39.732 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:20:39.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-21T01:20:39.733 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836484 2026-02-21T01:20:39.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T01:20:39.734 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:39.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836484 2026-02-21T01:20:39.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:20:40.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:20:40.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:41.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-21T01:20:41.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:20:41.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-21T01:20:41.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:20:41.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:20:41.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-21T01:20:41.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:20:41.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-21T01:20:41.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-21T01:20:41.163 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672964 2026-02-21T01:20:41.163 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:20:41.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-21T01:20:41.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:20:41.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:41.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:41.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:20:41.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:20:41.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:20:41.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:20:41.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:20:41.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:20:41.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:20:41.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:20:41.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:20:41.882 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:20:41.882 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:41.882 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:42.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: get_primary rbd GROUP 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-21T01:20:42.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-21T01:20:42.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local primary=1 2026-02-21T01:20:42.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: get_not_primary rbd GROUP 2026-02-21T01:20:42.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-21T01:20:42.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=GROUP 2026-02-21T01:20:42.383 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd GROUP 2026-02-21T01:20:42.383 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-21T01:20:42.383 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-21T01:20:42.383 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-21T01:20:42.383 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-21T01:20:42.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-21T01:20:42.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd GROUP 2026-02-21T01:20:42.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: local not_primary=0 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: test 0 '!=' 1 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: test 0 = 0 -o 0 = 1 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: teardown td/ceph-helpers 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:42.826 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:42.827 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:42.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:42.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:42.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:43.136 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:43.137 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:43.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:43.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:43.139 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:43.140 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:43.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:43.141 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:43.141 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:43.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:43.143 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:43.143 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:43.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:43.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:43.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:43.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:43.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:43.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:43.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:43.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_num_active_clean td/ceph-helpers 2026-02-21T01:20:43.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: test_get_num_active_clean: local dir=td/ceph-helpers 2026-02-21T01:20:43.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: test_get_num_active_clean: setup td/ceph-helpers 2026-02-21T01:20:43.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:43.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:43.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:43.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:43.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:43.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:43.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:43.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:43.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:43.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:43.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:43.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:43.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:43.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:43.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:43.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:43.163 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:43.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:43.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:43.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:43.163 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:43.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:43.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:43.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:43.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:43.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:43.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:43.170 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:43.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:43.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:43.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:43.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:43.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:43.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:43.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:43.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:43.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:43.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:43.222 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:43.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:43.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:43.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:43.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:43.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:43.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:43.282 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:43.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:43.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: test_get_num_active_clean: run_mgr td/ceph-helpers x 2026-02-21T01:20:43.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:43.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:43.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:43.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:43.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:43.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:43.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:43.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:43.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:43.453 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:43.453 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.453 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:43.454 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:43.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: test_get_num_active_clean: run_osd td/ceph-helpers 0 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:43.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:43.476 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:43.477 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:43.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:43.478 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85f8dd88-5c27-4dd3-8545-d7311e162a5f 2026-02-21T01:20:43.478 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 85f8dd88-5c27-4dd3-8545-d7311e162a5f' 2026-02-21T01:20:43.479 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 85f8dd88-5c27-4dd3-8545-d7311e162a5f 2026-02-21T01:20:43.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:43.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBrCJlpRfYCHRAAyapc112zCYWbl+Z8gRXVLg== 2026-02-21T01:20:43.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBrCJlpRfYCHRAAyapc112zCYWbl+Z8gRXVLg=="}' 2026-02-21T01:20:43.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85f8dd88-5c27-4dd3-8545-d7311e162a5f -i td/ceph-helpers/0/new.json 2026-02-21T01:20:43.596 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:43.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:43.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrCJlpRfYCHRAAyapc112zCYWbl+Z8gRXVLg== --osd-uuid 85f8dd88-5c27-4dd3-8545-d7311e162a5f 2026-02-21T01:20:43.622 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:43.618+0000 7fe16f188a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:43.624 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:43.618+0000 7fe16f188a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:43.625 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:43.622+0000 7fe16f188a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:43.626 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:43.622+0000 7fe16f188a40 -1 bdev(0x5580e6a3cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:43.626 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:43.622+0000 7fe16f188a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:44.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:44.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:44.341 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:44.341 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:44.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:44.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:44.453 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:44.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:44.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:44.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:44.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:44.469 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:44.466+0000 7f85ebc19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:44.470 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:44.466+0000 7f85ebc19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:44.471 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:44.466+0000 7f85ebc19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:44.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:44.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:44.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:45.050 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:45.046+0000 7f85ebc19a40 -1 Falling back to public interface 2026-02-21T01:20:45.681 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:45.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:45.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:45.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:45.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:45.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:45.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:46.008 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:46.002+0000 7f85ebc19a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:46.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:46.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:46.794 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:46.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:46.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:46.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:47.006 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4105616928,v1:127.0.0.1:6801/4105616928] [v2:127.0.0.1:6802/4105616928,v1:127.0.0.1:6803/4105616928] exists,up 85f8dd88-5c27-4dd3-8545-d7311e162a5f 2026-02-21T01:20:47.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:47.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:47.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:47.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1369: test_get_num_active_clean: create_rbd_pool 2026-02-21T01:20:47.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:20:47.213 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:20:47.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:20:47.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:20:47.448 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:20:47.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:20:48.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:20:48.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1370: test_get_num_active_clean: wait_for_clean 2026-02-21T01:20:48.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:20:48.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:20:48.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:20:48.738 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:20:48.738 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:20:48.738 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:20:48.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:20:48.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:20:48.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:20:48.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:20:48.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:20:48.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:20:48.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:20:48.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:20:48.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:20:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:20:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:20:49.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:20:49.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:20:49.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:20:49.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:20:49.135 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:20:49.135 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:20:49.135 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:49.135 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:20:49.136 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:20:49.136 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:49.137 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:20:49.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:20:49.138 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:20:49.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:20:49.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:49.359 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:20:49.359 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:20:50.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:20:50.361 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:50.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:20:50.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:20:50.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:50.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:50.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:20:50.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:20:50.841 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:20:50.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:20:50.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:20:50.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:20:50.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:20:50.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:20:51.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:20:51.049 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:20:51.050 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:51.050 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:51.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:20:51.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:20:51.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:20:51.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: get_num_active_clean 2026-02-21T01:20:51.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:20:51.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:20:51.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:20:51.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:20:51.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: local num_active_clean=4 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: test 4 = 4 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1373: test_get_num_active_clean: teardown td/ceph-helpers 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:51.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:51.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:51.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:51.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:51.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:51.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:51.657 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:51.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:51.659 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:51.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:51.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:51.661 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:51.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:51.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:51.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:51.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:51.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:51.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:51.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:51.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:51.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:51.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:51.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:51.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:20:51.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_num_active_or_peered td/ceph-helpers 2026-02-21T01:20:51.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1393: test_get_num_active_or_peered: local dir=td/ceph-helpers 2026-02-21T01:20:51.669 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: test_get_num_active_or_peered: setup td/ceph-helpers 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:20:51.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:20:51.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:20:51.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:20:51.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:20:51.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:20:51.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:20:51.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:20:51.675 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:20:51.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:51.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:20:51.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:20:51.676 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:20:51.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:20:51.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:20:51.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:20:51.679 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:20:51.679 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.679 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:20:51.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:20:51.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:20:51.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:20:51.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:20:51.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:20:51.684 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:20:51.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:20:51.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:20:51.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:20:51.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:51.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:20:51.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:20:51.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:20:51.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:20:51.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:20:51.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:51.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:20:51.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:51.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:51.708 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:51.708 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.708 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.708 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:51.708 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:20:51.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:20:51.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:20:51.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:51.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:51.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:20:51.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:20:51.749 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:51.749 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:51.749 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:51.750 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:51.750 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.750 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.750 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:51.750 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:51.750 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:20:51.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:20:51.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:20:51.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:20:51.806 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:20:51.807 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:20:51.807 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:20:51.807 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:20:51.807 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:20:51.807 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:20:51.808 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.808 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.808 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:20:51.808 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:20:51.808 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:20:51.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:51.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:51.980 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:51.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:20:51.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:20:52.000 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:20:52.001 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:20:52.002 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:20:52.003 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7be2d598-6cfc-42ef-a802-f75cbd7a2014 2026-02-21T01:20:52.003 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7be2d598-6cfc-42ef-a802-f75cbd7a2014' 2026-02-21T01:20:52.003 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 7be2d598-6cfc-42ef-a802-f75cbd7a2014 2026-02-21T01:20:52.003 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:20:52.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0CJlp1EDvABAAxtoNtm4xyB7v96wUeHB8UA== 2026-02-21T01:20:52.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0CJlp1EDvABAAxtoNtm4xyB7v96wUeHB8UA=="}' 2026-02-21T01:20:52.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7be2d598-6cfc-42ef-a802-f75cbd7a2014 -i td/ceph-helpers/0/new.json 2026-02-21T01:20:52.127 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:52.135 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:20:52.136 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0CJlp1EDvABAAxtoNtm4xyB7v96wUeHB8UA== --osd-uuid 7be2d598-6cfc-42ef-a802-f75cbd7a2014 2026-02-21T01:20:52.151 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:52.146+0000 7ffa998c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:52.153 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:52.150+0000 7ffa998c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:52.154 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:52.150+0000 7ffa998c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:52.155 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:52.150+0000 7ffa998c3a40 -1 bdev(0x564fb7934c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:20:52.155 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:52.150+0000 7ffa998c3a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:20:52.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:20:52.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:20:52.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:20:52.937 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:20:52.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:20:53.054 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:20:53.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:20:53.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:20:53.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:20:53.056 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:20:53.056 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:20:53.077 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:53.074+0000 7fdb200cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:53.081 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:53.078+0000 7fdb200cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:53.082 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:53.078+0000 7fdb200cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:20:53.166 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:20:53.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:53.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:53.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:53.410 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:53.406+0000 7fdb200cba40 -1 Falling back to public interface 2026-02-21T01:20:54.262 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:20:54.258+0000 7fdb200cba40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:20:54.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:54.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:54.276 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:20:54.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:20:54.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:54.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:54.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:20:55.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:20:55.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:20:55.491 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:20:55.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:20:55.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:20:55.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:20:55.707 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1306502841,v1:127.0.0.1:6801/1306502841] [v2:127.0.0.1:6802/1306502841,v1:127.0.0.1:6803/1306502841] exists,up 7be2d598-6cfc-42ef-a802-f75cbd7a2014 2026-02-21T01:20:55.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:20:55.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:20:55.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:20:55.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1399: test_get_num_active_or_peered: create_rbd_pool 2026-02-21T01:20:55.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:20:55.916 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:20:55.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:20:55.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:20:56.204 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:20:56.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:20:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:20:57.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1400: test_get_num_active_or_peered: wait_for_clean 2026-02-21T01:20:57.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:20:57.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:20:57.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:20:57.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:20:57.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:20:57.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:20:57.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:20:57.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:20:57.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:20:57.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:20:57.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:20:57.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:20:57.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:20:57.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:20:57.599 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:20:57.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:20:57.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:20:57.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:20:57.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:20:57.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:20:57.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:20:57.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:20:57.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:20:57.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:57.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:20:57.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:20:57.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:20:57.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:20:57.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:20:57.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:20:57.896 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:20:57.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:58.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:20:58.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:20:59.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:20:59.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:20:59.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:20:59.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:20:59.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:59.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:20:59.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:20:59.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:20:59.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:20:59.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:20:59.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:00.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:21:00.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:21:00.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:21:00.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: get_num_active_or_peered 2026-02-21T01:21:00.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-21T01:21:00.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-21T01:21:00.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-21T01:21:00.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-21T01:21:00.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-21T01:21:00.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: local num_peered=4 2026-02-21T01:21:00.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: test 4 = 4 2026-02-21T01:21:00.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: test_get_num_active_or_peered: teardown td/ceph-helpers 2026-02-21T01:21:00.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:00.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:00.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:00.434 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:00.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:00.435 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:00.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:00.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:00.437 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:00.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:00.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:00.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:00.438 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:00.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:00.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:00.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:00.443 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:00.443 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.443 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.443 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:00.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_num_pgs td/ceph-helpers 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1420: test_get_num_pgs: local dir=td/ceph-helpers 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1422: test_get_num_pgs: setup td/ceph-helpers 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:00.446 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:00.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:00.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:00.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:00.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:00.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:00.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:00.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:00.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:00.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:00.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:00.450 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:00.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:00.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:00.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:00.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:00.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:00.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:00.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:00.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:00.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:00.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:00.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:00.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:00.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:00.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:00.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:00.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:00.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:00.462 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.484 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:00.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:00.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:00.517 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.518 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.518 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:00.518 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:00.518 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:00.573 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:00.574 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:00.574 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.574 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.574 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:00.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:00.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:00.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1424: test_get_num_pgs: run_mgr td/ceph-helpers x 2026-02-21T01:21:00.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:00.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:00.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:00.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:00.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:00.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:00.747 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:00.747 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:00.747 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:00.747 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:00.747 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.747 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:00.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:00.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:00.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: test_get_num_pgs: run_osd td/ceph-helpers 0 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:00.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:00.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:00.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:00.772 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:00.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e0f9ff0-359b-47f5-849d-6f96b22b5b07 2026-02-21T01:21:00.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5e0f9ff0-359b-47f5-849d-6f96b22b5b07' 2026-02-21T01:21:00.773 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 5e0f9ff0-359b-47f5-849d-6f96b22b5b07 2026-02-21T01:21:00.773 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:00.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8CJlpTr6WLhAA+VYwMgjXvgctGR96R8JObg== 2026-02-21T01:21:00.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8CJlpTr6WLhAA+VYwMgjXvgctGR96R8JObg=="}' 2026-02-21T01:21:00.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e0f9ff0-359b-47f5-849d-6f96b22b5b07 -i td/ceph-helpers/0/new.json 2026-02-21T01:21:00.898 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:00.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:00.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8CJlpTr6WLhAA+VYwMgjXvgctGR96R8JObg== --osd-uuid 5e0f9ff0-359b-47f5-849d-6f96b22b5b07 2026-02-21T01:21:00.923 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:00.918+0000 7f2b97b77a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:00.924 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:00.922+0000 7f2b97b77a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:00.925 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:00.922+0000 7f2b97b77a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:00.926 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:00.922+0000 7f2b97b77a40 -1 bdev(0x555e262c4c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:00.926 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:00.922+0000 7f2b97b77a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:01.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:01.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:01.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:01.664 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:01.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:01.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:01.784 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:01.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:01.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:01.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:01.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:01.809 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:01.806+0000 7fbd3fe34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:01.811 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:01.806+0000 7fbd3fe34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:01.812 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:01.810+0000 7fbd3fe34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:01.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:01.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:01.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:01.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:01.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:01.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:01.893 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:01.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:01.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:01.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:01.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:02.302 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:02.298+0000 7fbd3fe34a40 -1 Falling back to public interface 2026-02-21T01:21:02.998 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:02.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:02.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:02.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:02.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:03.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:03.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:03.282 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:03.278+0000 7fbd3fe34a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:21:04.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:04.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:04.119 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:04.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:04.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:04.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:04.331 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3751095849,v1:127.0.0.1:6801/3751095849] [v2:127.0.0.1:6802/3751095849,v1:127.0.0.1:6803/3751095849] exists,up 5e0f9ff0-359b-47f5-849d-6f96b22b5b07 2026-02-21T01:21:04.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:04.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:04.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:04.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1426: test_get_num_pgs: create_rbd_pool 2026-02-21T01:21:04.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:21:04.539 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:21:04.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:21:04.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:21:04.776 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:21:04.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:21:05.786 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:21:06.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1427: test_get_num_pgs: wait_for_clean 2026-02-21T01:21:06.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:06.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:06.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:21:06.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:21:06.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:21:06.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:21:06.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:21:06.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:21:06.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:21:06.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:21:06.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:21:06.380 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:21:06.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:21:06.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:21:06.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:21:06.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:21:06.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:06.459 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:21:06.461 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:21:06.461 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:06.461 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:21:06.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:21:06.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:21:06.463 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:21:06.463 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:06.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:21:06.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:21:07.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:21:07.679 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:07.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:21:07.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:21:07.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:07.895 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:08.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:21:08.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:21:08.179 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:21:08.179 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:21:08.180 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:21:08.180 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:21:08.180 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:21:08.180 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:21:08.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:21:08.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:21:08.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:08.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:08.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:21:08.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:21:08.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:21:08.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: get_num_pgs 2026-02-21T01:21:08.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:08.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: local num_pgs=4 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: test 4 -gt 0 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1430: test_get_num_pgs: teardown td/ceph-helpers 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:08.963 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:08.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:08.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:08.964 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:09.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:09.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:09.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:09.073 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:09.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:09.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:09.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:09.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:09.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:09.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:09.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:09.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:09.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:09.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:09.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:09.085 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.085 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_op_scheduler td/ceph-helpers 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: test_get_op_scheduler: local dir=td/ceph-helpers 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: test_get_op_scheduler: setup td/ceph-helpers 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:09.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:09.089 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:09.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:09.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:09.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:09.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:09.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:09.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:09.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:09.092 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:09.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:09.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:09.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:09.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:09.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:09.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:09.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:09.097 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:09.097 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.097 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:09.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:09.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:09.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:09.100 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:09.100 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.100 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:09.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: test_get_op_scheduler: run_mon td/ceph-helpers a 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:09.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:09.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:09.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:09.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:21:09.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:09.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:09.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:09.124 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:09.124 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.124 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.124 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:09.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:21:09.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:09.153 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:09.154 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:09.154 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.154 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.154 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:09.154 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:09.154 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:09.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:09.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:09.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:09.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:09.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:09.210 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:09.211 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: test_get_op_scheduler: run_mgr td/ceph-helpers x 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:09.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:09.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:09.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:09.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:09.382 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:09.382 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.382 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:09.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:09.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:09.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:09.402 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=wpq 2026-02-21T01:21:09.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:09.404 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:09.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a9b416b3-49f1-4113-8506-6775a033694d 2026-02-21T01:21:09.404 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 a9b416b3-49f1-4113-8506-6775a033694d 2026-02-21T01:21:09.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a9b416b3-49f1-4113-8506-6775a033694d' 2026-02-21T01:21:09.404 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:09.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFCJlp5fm4GBAAQ2Mv7BivPd/IGAPTQA3vAg== 2026-02-21T01:21:09.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFCJlp5fm4GBAAQ2Mv7BivPd/IGAPTQA3vAg=="}' 2026-02-21T01:21:09.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a9b416b3-49f1-4113-8506-6775a033694d -i td/ceph-helpers/0/new.json 2026-02-21T01:21:09.529 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:09.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:09.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQCFCJlp5fm4GBAAQ2Mv7BivPd/IGAPTQA3vAg== --osd-uuid a9b416b3-49f1-4113-8506-6775a033694d 2026-02-21T01:21:09.555 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:09.549+0000 7ff484ef9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:09.557 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:09.553+0000 7ff484ef9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:09.557 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:09.553+0000 7ff484ef9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:09.558 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:09.553+0000 7ff484ef9a40 -1 bdev(0x563fbcee0c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:09.558 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:09.553+0000 7ff484ef9a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:10.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:10.276 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:10.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:10.277 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:10.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:10.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:10.400 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:10.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2026-02-21T01:21:10.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:10.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:10.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:10.426 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:10.421+0000 7f0919853a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:10.428 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:10.421+0000 7f0919853a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:10.429 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:10.425+0000 7f0919853a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:10.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:10.509 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:10.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:10.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:10.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:10.854 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:10.849+0000 7f0919853a40 -1 Falling back to public interface 2026-02-21T01:21:11.461 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:11.457+0000 7f0919853a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:21:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:11.623 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:11.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:12.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:12.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:12.736 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:12.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:12.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:12.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:12.946 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/886259843,v1:127.0.0.1:6801/886259843] [v2:127.0.0.1:6802/886259843,v1:127.0.0.1:6803/886259843] exists,up a9b416b3-49f1-4113-8506-6775a033694d 2026-02-21T01:21:12.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: test_get_op_scheduler: get_op_scheduler 0 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: get_op_scheduler: local id=0 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: get_op_scheduler: get_config osd 0 osd_op_queue 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-21T01:21:12.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:12.948 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:21:12.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:12.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_op_queue 2026-02-21T01:21:13.014 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: test_get_op_scheduler: test wpq = wpq 2026-02-21T01:21:13.014 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:21:13.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:13.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T01:21:13.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:21:13.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:13.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=760e578c-661f-4a42-b8ef-f89ba6b43f95 2026-02-21T01:21:13.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 760e578c-661f-4a42-b8ef-f89ba6b43f95' 2026-02-21T01:21:13.019 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 760e578c-661f-4a42-b8ef-f89ba6b43f95 2026-02-21T01:21:13.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:13.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCJCJlpvFWaARAAZkkohKhyKNh1gtN58W/Rmg== 2026-02-21T01:21:13.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCJCJlpvFWaARAAZkkohKhyKNh1gtN58W/Rmg=="}' 2026-02-21T01:21:13.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 760e578c-661f-4a42-b8ef-f89ba6b43f95 -i td/ceph-helpers/1/new.json 2026-02-21T01:21:13.240 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:13.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:21:13.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCJCJlpvFWaARAAZkkohKhyKNh1gtN58W/Rmg== --osd-uuid 760e578c-661f-4a42-b8ef-f89ba6b43f95 2026-02-21T01:21:13.266 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:13.261+0000 7f8532f2ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:13.267 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:13.261+0000 7f8532f2ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:13.268 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:13.265+0000 7f8532f2ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:13.269 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:13.265+0000 7f8532f2ba40 -1 bdev(0x55c1f7b9cc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:13.269 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:13.265+0000 7f8532f2ba40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:21:13.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:21:13.961 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:13.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:21:13.962 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:21:13.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:14.249 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:21:14.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:21:14.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T01:21:14.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:14.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:14.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:14.274 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:14.269+0000 7fe386882a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:14.276 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:14.269+0000 7fe386882a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:14.277 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:14.273+0000 7fe386882a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:14.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:21:14.650 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:14.645+0000 7fe386882a40 -1 Falling back to public interface 2026-02-21T01:21:14.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:15.202 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:15.197+0000 7fe386882a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:21:15.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:15.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:15.676 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:15.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:15.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:15.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:21:15.882 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/594086623,v1:127.0.0.1:6811/594086623] [v2:127.0.0.1:6812/594086623,v1:127.0.0.1:6813/594086623] exists,up 760e578c-661f-4a42-b8ef-f89ba6b43f95 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: test_get_op_scheduler: get_op_scheduler 1 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: get_op_scheduler: local id=1 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: get_op_scheduler: get_config osd 1 osd_op_queue 2026-02-21T01:21:15.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-21T01:21:15.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=1 2026-02-21T01:21:15.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-21T01:21:15.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-21T01:21:15.885 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.1 2026-02-21T01:21:15.885 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-21T01:21:15.885 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-21T01:21:15.886 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:15.886 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:15.886 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:15.886 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.1.asok 2026-02-21T01:21:15.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:15.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-osd.1.asok config get osd_op_queue 2026-02-21T01:21:15.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2026-02-21T01:21:15.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: test_get_op_scheduler: teardown td/ceph-helpers 2026-02-21T01:21:15.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:15.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:15.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:15.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:15.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:15.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:15.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:15.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:16.056 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:16.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:16.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:16.057 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:16.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:16.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:16.058 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:16.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:16.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:16.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:16.060 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:16.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:16.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:16.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:16.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:16.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.069 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1450: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1452: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:16.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:16.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:16.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:16.073 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:16.074 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:16.074 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:16.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:16.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:16.076 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:16.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:16.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:16.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:16.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:16.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:16.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:16.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:16.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:16.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:16.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:16.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:16.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:16.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:16.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1453: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:16.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:16.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:16.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:16.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:16.108 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:16.108 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.108 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:16.109 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:16.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:16.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:16.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:16.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:16.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:16.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:16.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:16.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:16.196 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:16.196 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:16.197 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:16.197 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:16.197 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:16.198 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:16.198 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:16.198 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:16.198 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:16.198 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.198 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.199 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:16.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:16.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:16.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1454: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-21T01:21:16.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:16.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:16.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:16.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:16.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:16.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:16.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:16.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:16.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:16.369 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:16.369 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.369 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.369 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:16.369 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:16.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:16.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:16.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:16.390 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:16.390 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:16.390 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:16.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:16.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:16.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:16.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:16.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:16.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bda6d653-ced2-455c-b004-7222c0a8ebc4 2026-02-21T01:21:16.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bda6d653-ced2-455c-b004-7222c0a8ebc4' 2026-02-21T01:21:16.393 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 bda6d653-ced2-455c-b004-7222c0a8ebc4 2026-02-21T01:21:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:16.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMCJlpkaAgGBAAbaZg9O+NPWJUhRUlNUeMnA== 2026-02-21T01:21:16.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMCJlpkaAgGBAAbaZg9O+NPWJUhRUlNUeMnA=="}' 2026-02-21T01:21:16.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bda6d653-ced2-455c-b004-7222c0a8ebc4 -i td/ceph-helpers/0/new.json 2026-02-21T01:21:16.515 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:16.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:16.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMCJlpkaAgGBAAbaZg9O+NPWJUhRUlNUeMnA== --osd-uuid bda6d653-ced2-455c-b004-7222c0a8ebc4 2026-02-21T01:21:16.539 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:16.533+0000 7f973e083a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:16.541 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:16.537+0000 7f973e083a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:16.542 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:16.537+0000 7f973e083a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:16.542 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:16.537+0000 7f973e083a40 -1 bdev(0x555a7f84cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:16.542 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:16.537+0000 7f973e083a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:17.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:17.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:17.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:17.237 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:17.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:17.347 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:17.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:17.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:17.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:17.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:17.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:17.362 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:17.357+0000 7fc7b41a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:17.364 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:17.357+0000 7fc7b41a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:17.365 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:17.361+0000 7fc7b41a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:17.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:17.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:17.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:17.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:17.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:17.777 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:17.773+0000 7fc7b41a5a40 -1 Falling back to public interface 2026-02-21T01:21:18.422 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:18.417+0000 7fc7b41a5a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:21:18.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:18.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:18.569 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:18.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:18.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:18.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:18.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:19.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:19.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:19.706 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:19.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:19.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:19.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:19.925 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3986645696,v1:127.0.0.1:6801/3986645696] [v2:127.0.0.1:6802/3986645696,v1:127.0.0.1:6803/3986645696] exists,up bda6d653-ced2-455c-b004-7222c0a8ebc4 2026-02-21T01:21:19.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:19.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:19.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:19.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1456: test_get_osd_id_used_by_pgs: create_rbd_pool 2026-02-21T01:21:19.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:21:20.126 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:21:20.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:21:20.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:21:20.359 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:21:20.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:21:21.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:21:21.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1457: test_get_osd_id_used_by_pgs: wait_for_clean 2026-02-21T01:21:21.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:21:21.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:21:21.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:21:21.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:21:21.653 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:21.653 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:21.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:21.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:21.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:21.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:21:21.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:21:21.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:21:21.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:21:21.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:21:21.752 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:21:21.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:21:21.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:21:21.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:21:21.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:21:22.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:21:22.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:21:22.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:21:22.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:21:22.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:22.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:21:22.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:21:22.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:22.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:21:22.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:21:22.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:21:22.058 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:21:22.058 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:22.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:21:22.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:21:23.273 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:21:23.274 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:23.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:21:23.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:21:23.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:23.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:23.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:21:23.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:21:23.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:21:23.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:21:23.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:21:23.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:21:23.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:21:23.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:21:23.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:21:23.986 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:21:23.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:23.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:24.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:21:24.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:21:24.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:21:24.273 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-21T01:21:24.273 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: uniq 2026-02-21T01:21:24.273 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-21T01:21:24.273 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-21T01:21:24.273 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: local osd_ids=0 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: test 0 = 0 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1460: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:24.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:24.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:24.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:24.597 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:24.597 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:24.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:24.598 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:24.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:24.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:24.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:24.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:24.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:24.602 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:24.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:24.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:24.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:24.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:24.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:24.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_osds td/ceph-helpers 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: test_get_osds: local dir=td/ceph-helpers 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1024: test_get_osds: setup td/ceph-helpers 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:24.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:24.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:24.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:24.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:24.612 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:24.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:24.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:24.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:24.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:24.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:24.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:24.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:24.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:24.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:24.616 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:24.617 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:24.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:24.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:24.619 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:24.620 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.620 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:24.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:24.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:24.621 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:24.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:24.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.622 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:24.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:24.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-21T01:21:24.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:24.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-21T01:21:24.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:24.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:24.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:24.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:24.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.682 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.683 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:24.683 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:24.683 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:24.742 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:24.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:24.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:24.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:24.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:24.743 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:24.743 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:24.744 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1026: test_get_osds: run_mgr td/ceph-helpers x 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:24.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:24.911 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:24.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:24.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:24.912 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:24.912 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.912 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:24.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:24.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: test_get_osds: run_osd td/ceph-helpers 0 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:24.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:24.936 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:24.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:24.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:24.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:24.938 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:24.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=13cbb7ad-7fa2-413d-803c-d862978f594b 2026-02-21T01:21:24.939 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 13cbb7ad-7fa2-413d-803c-d862978f594b 2026-02-21T01:21:24.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 13cbb7ad-7fa2-413d-803c-d862978f594b' 2026-02-21T01:21:24.939 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:24.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCUCJlpZ6xyOBAAITH/sBXS9fY/iS1R8lX+Ig== 2026-02-21T01:21:24.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCUCJlpZ6xyOBAAITH/sBXS9fY/iS1R8lX+Ig=="}' 2026-02-21T01:21:24.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 13cbb7ad-7fa2-413d-803c-d862978f594b -i td/ceph-helpers/0/new.json 2026-02-21T01:21:25.059 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:25.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:25.069 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUCJlpZ6xyOBAAITH/sBXS9fY/iS1R8lX+Ig== --osd-uuid 13cbb7ad-7fa2-413d-803c-d862978f594b 2026-02-21T01:21:25.085 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:25.081+0000 7f7cfdd5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:25.087 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:25.081+0000 7f7cfdd5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:25.088 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:25.085+0000 7f7cfdd5da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:25.089 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:25.085+0000 7f7cfdd5da40 -1 bdev(0x5639eac7cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:25.089 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:25.085+0000 7f7cfdd5da40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:25.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:25.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:25.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:25.956 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:25.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:26.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:26.078 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:26.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:26.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:26.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:26.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:26.101 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:26.097+0000 7f0f8316ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:26.103 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:26.097+0000 7f0f8316ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:26.104 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:26.101+0000 7f0f8316ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:26.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:26.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:26.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:26.546 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:26.541+0000 7f0f8316ea40 -1 Falling back to public interface 2026-02-21T01:21:27.262 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:27.257+0000 7f0f8316ea40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:21:27.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:27.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:27.299 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:27.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:27.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:27.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:27.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:28.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:28.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:28.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:28.514 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:28.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:28.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:28.729 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4168940176,v1:127.0.0.1:6801/4168940176] [v2:127.0.0.1:6802/4168940176,v1:127.0.0.1:6803/4168940176] exists,up 13cbb7ad-7fa2-413d-803c-d862978f594b 2026-02-21T01:21:28.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:28.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1028: test_get_osds: run_osd td/ceph-helpers 1 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:28.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:28.735 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:28.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1d358e0f-93c8-4468-9d6a-254ceb78b81b 2026-02-21T01:21:28.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1d358e0f-93c8-4468-9d6a-254ceb78b81b' 2026-02-21T01:21:28.738 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 1d358e0f-93c8-4468-9d6a-254ceb78b81b 2026-02-21T01:21:28.738 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:28.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYCJlpeGZ4LBAAGxv6XT7NDLjrh6LrnIhwPA== 2026-02-21T01:21:28.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYCJlpeGZ4LBAAGxv6XT7NDLjrh6LrnIhwPA=="}' 2026-02-21T01:21:28.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1d358e0f-93c8-4468-9d6a-254ceb78b81b -i td/ceph-helpers/1/new.json 2026-02-21T01:21:28.962 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:28.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:21:28.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYCJlpeGZ4LBAAGxv6XT7NDLjrh6LrnIhwPA== --osd-uuid 1d358e0f-93c8-4468-9d6a-254ceb78b81b 2026-02-21T01:21:28.990 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:28.985+0000 7fac9d1eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:28.991 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:28.985+0000 7fac9d1eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:28.992 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:28.989+0000 7fac9d1eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:28.993 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:28.989+0000 7fac9d1eca40 -1 bdev(0x557e718eac00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:28.993 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:28.989+0000 7fac9d1eca40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:21:29.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:21:29.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:29.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:21:29.620 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:21:29.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:29.910 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:21:29.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:21:29.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:29.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:29.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:29.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:29.926 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:29.921+0000 7f9cdb6d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:29.927 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:29.921+0000 7f9cdb6d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:29.928 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:29.925+0000 7f9cdb6d7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:30.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:21:30.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:30.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:21:30.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:30.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:30.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:30.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:30.114 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:30.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:30.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:21:30.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:30.378 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:30.373+0000 7f9cdb6d7a40 -1 Falling back to public interface 2026-02-21T01:21:31.054 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:31.049+0000 7f9cdb6d7a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:21:31.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:31.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:31.335 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:31.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:31.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:31.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:21:31.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:32.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:32.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:32.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:32.549 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:32.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:32.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:21:32.766 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2084390681,v1:127.0.0.1:6811/2084390681] [v2:127.0.0.1:6812/2084390681,v1:127.0.0.1:6813/2084390681] exists,up 1d358e0f-93c8-4468-9d6a-254ceb78b81b 2026-02-21T01:21:32.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:32.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:32.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:32.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: test_get_osds: create_rbd_pool 2026-02-21T01:21:32.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:21:32.974 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:21:32.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:21:32.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:21:33.262 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:21:33.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:21:34.274 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:21:34.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: test_get_osds: wait_for_clean 2026-02-21T01:21:34.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:21:34.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:21:34.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:21:34.598 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:21:34.599 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:34.599 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:34.599 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:34.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:34.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:34.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:21:34.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:21:34.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:21:34.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:21:34.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:21:34.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:21:34.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:21:34.909 INFO:tasks.workunit.client.0.trial106.stderr:1' 2026-02-21T01:21:34.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:21:34.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:21:34.910 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:21:34.989 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-21T01:21:34.989 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-21T01:21:34.989 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T01:21:34.989 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:21:34.989 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:21:35.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-21T01:21:35.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-21T01:21:35.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-21T01:21:35.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:21:35.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-21T01:21:35.067 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:21:35.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:21:35.068 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-21T01:21:35.069 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:21:35.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-21T01:21:35.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T01:21:35.070 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836484 2026-02-21T01:21:35.070 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:35.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-21T01:21:35.294 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:21:35.295 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-21T01:21:35.295 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:21:35.296 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:21:35.297 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-21T01:21:35.297 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:21:35.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-21T01:21:35.298 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 38654705667 2026-02-21T01:21:35.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-21T01:21:35.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:21:35.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-21T01:21:35.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:21:35.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:35.522 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:35.812 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:21:35.812 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:21:35.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:21:35.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:21:35.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:21:35.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:21:35.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:21:35.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:21:36.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:21:36.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:21:36.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:36.024 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:36.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:21:36.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:21:36.298 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:21:36.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: create_rbd_pool 2026-02-21T01:21:36.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:21:36.512 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:21:36.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:21:36.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:21:36.787 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:21:36.798 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:21:37.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:21:38.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: get_osds rbd GROUP 2026-02-21T01:21:38.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=rbd 2026-02-21T01:21:38.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=GROUP 2026-02-21T01:21:38.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2026-02-21T01:21:38.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map rbd GROUP 2026-02-21T01:21:38.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-21T01:21:38.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-21T01:21:38.299 INFO:tasks.workunit.client.0.trial106.stderr:0' 2026-02-21T01:21:38.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-21T01:21:38.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: teardown td/ceph-helpers 2026-02-21T01:21:38.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:38.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:38.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:38.409 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:38.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:38.410 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:38.412 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:38.412 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:38.412 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:38.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:38.413 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:38.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:38.414 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:38.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:38.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:38.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:38.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:38.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_pg td/ceph-helpers 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1091: test_get_pg: local dir=td/ceph-helpers 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: test_get_pg: setup td/ceph-helpers 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:38.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:38.425 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:38.425 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:38.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:38.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:38.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:38.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:38.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:38.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:38.428 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:38.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:38.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:38.430 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:38.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:38.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:38.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:38.432 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:38.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:38.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:38.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:38.435 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:38.435 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.435 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:38.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:38.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:38.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:38.437 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:38.438 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.438 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:38.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:38.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:38.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:38.463 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:38.464 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:38.464 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:38.464 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:38.464 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.464 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.464 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:38.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:38.493 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:38.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:38.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:38.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:38.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.495 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:38.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:38.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:38.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:38.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:38.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:38.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:38.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:38.555 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:38.555 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:38.555 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:38.555 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:38.556 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.556 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.556 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:38.556 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:38.556 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1095: test_get_pg: run_mgr td/ceph-helpers x 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:38.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:38.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:38.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:38.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:38.727 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:38.727 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.727 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:38.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:38.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: test_get_pg: run_osd td/ceph-helpers 0 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:38.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:38.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:38.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:38.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aa13ac7a-fc2d-445a-960f-534ca003c8f0 2026-02-21T01:21:38.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 aa13ac7a-fc2d-445a-960f-534ca003c8f0' 2026-02-21T01:21:38.749 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 aa13ac7a-fc2d-445a-960f-534ca003c8f0 2026-02-21T01:21:38.749 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:38.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCiCJlpI8pULRAA78Wd3hzoUomnIIvm62W3kQ== 2026-02-21T01:21:38.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCiCJlpI8pULRAA78Wd3hzoUomnIIvm62W3kQ=="}' 2026-02-21T01:21:38.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aa13ac7a-fc2d-445a-960f-534ca003c8f0 -i td/ceph-helpers/0/new.json 2026-02-21T01:21:38.870 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:38.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:38.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiCJlpI8pULRAA78Wd3hzoUomnIIvm62W3kQ== --osd-uuid aa13ac7a-fc2d-445a-960f-534ca003c8f0 2026-02-21T01:21:38.895 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:38.889+0000 7fa2a2a49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:38.896 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:38.893+0000 7fa2a2a49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:38.897 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:38.893+0000 7fa2a2a49a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:38.897 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:38.893+0000 7fa2a2a49a40 -1 bdev(0x561f5df5ac00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:38.897 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:38.893+0000 7fa2a2a49a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:39.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:39.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:39.607 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:39.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:39.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:39.718 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:39.718 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:39.718 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:39.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:39.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:39.719 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:39.733 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:39.729+0000 7fdf22eeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:39.735 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:39.729+0000 7fdf22eeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:39.736 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:39.733+0000 7fdf22eeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:39.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:39.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:40.217 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:40.213+0000 7fdf22eeea40 -1 Falling back to public interface 2026-02-21T01:21:40.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:40.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:40.933 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:40.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:40.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:40.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:41.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:41.241 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:41.237+0000 7fdf22eeea40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:21:42.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:42.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:42.063 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:42.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:42.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:42.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:42.279 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1101586239,v1:127.0.0.1:6801/1101586239] [v2:127.0.0.1:6802/1101586239,v1:127.0.0.1:6803/1101586239] exists,up aa13ac7a-fc2d-445a-960f-534ca003c8f0 2026-02-21T01:21:42.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:42.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:42.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:42.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1097: test_get_pg: create_rbd_pool 2026-02-21T01:21:42.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:21:42.485 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:21:42.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:21:42.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:21:42.727 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:21:42.736 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:21:43.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: test_get_pg: wait_for_clean 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:44.018 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:44.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:44.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:44.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:44.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:21:44.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:21:44.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:21:44.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:21:44.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:21:44.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:21:44.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:21:44.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:21:44.338 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:21:44.339 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:21:44.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:21:44.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:21:44.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:21:44.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:21:44.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:44.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:21:44.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:21:44.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:44.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:21:44.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:21:44.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:21:44.422 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:21:44.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:44.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:21:44.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:21:45.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:21:45.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:45.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:21:45.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:21:45.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:45.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:46.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:21:46.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:21:46.144 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:21:46.144 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:21:46.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:21:46.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:21:46.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:21:46.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:21:46.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:21:46.356 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:21:46.356 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:46.356 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:46.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:21:46.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: get_pg rbd GROUP 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=GROUP 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd GROUP 2026-02-21T01:21:46.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-21T01:21:46.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: teardown td/ceph-helpers 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:46.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:46.967 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:46.968 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:46.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:46.969 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:46.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:46.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:46.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:46.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:46.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:46.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:46.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:46.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:46.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:46.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:46.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:46.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:46.979 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:46.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_primary td/ceph-helpers 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: test_get_primary: local dir=td/ceph-helpers 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: test_get_primary: setup td/ceph-helpers 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:46.981 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:46.982 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:46.982 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:46.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:46.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:46.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:46.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:46.984 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:46.985 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:46.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:46.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:46.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:46.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:46.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:46.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:46.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:46.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:46.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:46.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:46.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:46.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:46.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:46.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:46.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:46.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:46.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:46.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:46.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:46.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:46.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:46.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:46.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:46.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:47.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:47.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:47.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:47.019 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:47.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:47.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:47.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:47.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:47.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:47.052 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:47.053 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:47.053 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:47.053 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:47.053 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:47.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:47.053 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:47.110 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:47.110 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:47.110 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:47.110 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:47.110 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:47.110 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:47.111 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:47.111 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:47.111 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:47.111 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:47.111 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:47.111 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:47.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:47.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: test_get_primary: local osd=0 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: test_get_primary: run_mgr td/ceph-helpers x 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:47.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:47.279 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:47.280 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:47.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: test_get_primary: run_osd td/ceph-helpers 0 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:47.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:47.300 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:47.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:47.302 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:47.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8fa43385-8580-43b0-a4df-c7e24d0237a5 2026-02-21T01:21:47.303 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 8fa43385-8580-43b0-a4df-c7e24d0237a5 2026-02-21T01:21:47.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8fa43385-8580-43b0-a4df-c7e24d0237a5' 2026-02-21T01:21:47.303 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:47.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrCJlpvT7AEhAAfNW3DoL8zotd+J9qlDXk9g== 2026-02-21T01:21:47.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrCJlpvT7AEhAAfNW3DoL8zotd+J9qlDXk9g=="}' 2026-02-21T01:21:47.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8fa43385-8580-43b0-a4df-c7e24d0237a5 -i td/ceph-helpers/0/new.json 2026-02-21T01:21:47.422 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:47.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:47.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrCJlpvT7AEhAAfNW3DoL8zotd+J9qlDXk9g== --osd-uuid 8fa43385-8580-43b0-a4df-c7e24d0237a5 2026-02-21T01:21:47.447 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:47.441+0000 7fe763370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:47.448 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:47.445+0000 7fe763370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:47.449 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:47.445+0000 7fe763370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:47.449 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:47.445+0000 7fe763370a40 -1 bdev(0x562bf60eec00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:47.450 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:47.445+0000 7fe763370a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:48.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:48.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:48.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:48.164 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:48.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:48.277 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:48.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:48.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:48.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:48.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:48.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:48.293 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:48.289+0000 7ff935ac8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:48.294 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:48.289+0000 7ff935ac8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:48.295 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:48.289+0000 7ff935ac8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:48.392 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:48.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:48.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:48.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:48.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:48.693 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:48.689+0000 7ff935ac8a40 -1 Falling back to public interface 2026-02-21T01:21:49.350 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:49.345+0000 7ff935ac8a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:21:49.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:49.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:49.502 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:49.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:49.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:49.506 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:49.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:50.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:50.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:50.648 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:21:50.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:21:50.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:50.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:50.864 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3627522403,v1:127.0.0.1:6801/3627522403] [v2:127.0.0.1:6802/3627522403,v1:127.0.0.1:6803/3627522403] exists,up 8fa43385-8580-43b0-a4df-c7e24d0237a5 2026-02-21T01:21:50.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:21:50.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:21:50.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:21:50.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: test_get_primary: create_rbd_pool 2026-02-21T01:21:50.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:21:51.073 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:21:51.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:21:51.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:21:51.324 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:21:51.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:21:52.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:21:52.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: test_get_primary: wait_for_clean 2026-02-21T01:21:52.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:21:52.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:21:52.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:21:52.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:21:52.663 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:52.663 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:52.663 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:52.663 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:52.663 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:52.764 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:21:52.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:21:52.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:21:52.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:21:52.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:21:52.765 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:21:52.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:21:52.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:21:52.982 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:21:52.983 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:21:53.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:21:53.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:21:53.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:21:53.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:21:53.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:53.063 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:21:53.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:21:53.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:21:53.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:21:53.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:21:53.065 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:21:53.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:21:53.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:53.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:21:53.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:21:54.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:21:54.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:21:54.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:21:54.498 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:21:54.498 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:54.498 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:21:54.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:21:54.782 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:21:54.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:21:54.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:21:54.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:21:54.985 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: get_primary rbd GROUP 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-21T01:21:55.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-21T01:21:55.270 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-21T01:21:55.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: test 0 = 0 2026-02-21T01:21:55.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: test_get_primary: teardown td/ceph-helpers 2026-02-21T01:21:55.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:55.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:55.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:55.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:55.486 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:55.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:55.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:55.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:55.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:55.594 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:55.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:55.595 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:55.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:55.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:55.597 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:55.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:55.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:55.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:55.599 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:55.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:55.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:55.601 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:55.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:55.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:55.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:55.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:55.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:55.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:55.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:55.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_get_timeout_delays td/ceph-helpers 2026-02-21T01:21:55.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 2026-02-21T01:21:55.607 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.607 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '1 ' = '1 ' 2026-02-21T01:21:55.621 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 5 2026-02-21T01:21:55.621 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.622 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.622 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2026-02-21T01:21:55.647 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 6 2026-02-21T01:21:55.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.648 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.648 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2026-02-21T01:21:55.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 7 2026-02-21T01:21:55.673 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.673 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2026-02-21T01:21:55.699 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 8 2026-02-21T01:21:55.700 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.700 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2026-02-21T01:21:55.732 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 1 .1 2026-02-21T01:21:55.733 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.733 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.733 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.733 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.733 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2026-02-21T01:21:55.768 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 1.5 .1 2026-02-21T01:21:55.768 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.769 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.769 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.769 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.769 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.808 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2026-02-21T01:21:55.808 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 5 .1 2026-02-21T01:21:55.809 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.810 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.810 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.851 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2026-02-21T01:21:55.852 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: get_timeout_delays 6 .1 2026-02-21T01:21:55.852 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.852 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.853 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2026-02-21T01:21:55.904 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: get_timeout_delays 6.3 .1 2026-02-21T01:21:55.904 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.905 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:55.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2026-02-21T01:21:55.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: get_timeout_delays 20 .1 2026-02-21T01:21:55.956 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:55.956 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:55.957 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:55.957 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:55.957 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:56.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2026-02-21T01:21:56.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: get_timeout_delays 300 .1 0 2026-02-21T01:21:56.025 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:56.025 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:56.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:56.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:56.025 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:56.128 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2026-02-21T01:21:56.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: get_timeout_delays 300 .1 10 2026-02-21T01:21:56.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:21:56.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:21:56.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:21:56.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:21:56.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:21:56.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_is_clean td/ceph-helpers 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: test_is_clean: local dir=td/ceph-helpers 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: test_is_clean: setup td/ceph-helpers 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:21:56.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:21:56.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:21:56.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:21:56.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:21:56.420 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:21:56.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:21:56.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:21:56.421 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:21:56.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:56.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:21:56.422 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:21:56.422 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:21:56.424 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:21:56.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:21:56.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:21:56.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:21:56.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:21:56.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:21:56.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:21:56.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:21:56.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:21:56.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:21:56.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:21:56.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:56.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:21:56.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:56.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:56.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:56.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:56.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:21:56.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:21:56.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:21:56.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:56.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:56.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:21:56.486 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:21:56.486 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:56.486 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:56.486 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:56.487 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:56.487 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.487 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.487 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:56.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:56.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:21:56.545 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:21:56.546 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:21:56.546 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.547 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:21:56.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:21:56.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: test_is_clean: run_mgr td/ceph-helpers x 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:21:56.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:21:56.718 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:21:56.719 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:56.719 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:56.719 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:56.719 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.719 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.720 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:56.720 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:56.720 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:21:56.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: test_is_clean: run_osd td/ceph-helpers 0 2026-02-21T01:21:56.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:21:56.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:21:56.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:21:56.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:21:56.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:21:56.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:21:56.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:21:56.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:21:56.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a0e58c06-dc32-419a-9bcd-7f82e1ca72f9 2026-02-21T01:21:56.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a0e58c06-dc32-419a-9bcd-7f82e1ca72f9' 2026-02-21T01:21:56.748 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 a0e58c06-dc32-419a-9bcd-7f82e1ca72f9 2026-02-21T01:21:56.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:21:56.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0CJlp/ZNdLRAAI1LkNtphJ+eS82ybwzVLeA== 2026-02-21T01:21:56.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0CJlp/ZNdLRAAI1LkNtphJ+eS82ybwzVLeA=="}' 2026-02-21T01:21:56.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a0e58c06-dc32-419a-9bcd-7f82e1ca72f9 -i td/ceph-helpers/0/new.json 2026-02-21T01:21:56.873 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:56.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:21:56.882 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0CJlp/ZNdLRAAI1LkNtphJ+eS82ybwzVLeA== --osd-uuid a0e58c06-dc32-419a-9bcd-7f82e1ca72f9 2026-02-21T01:21:56.898 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:56.893+0000 7fc88f4dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:56.899 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:56.897+0000 7fc88f4dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:56.900 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:56.897+0000 7fc88f4dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:56.901 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:56.897+0000 7fc88f4dea40 -1 bdev(0x561177326c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:21:56.901 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:56.897+0000 7fc88f4dea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:21:57.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:21:57.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:21:57.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:21:57.600 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:21:57.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:21:57.726 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:21:57.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:21:57.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:21:57.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:21:57.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:21:57.728 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:21:57.750 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:57.745+0000 7fdf01759a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:57.752 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:57.749+0000 7fdf01759a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:57.754 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:57.749+0000 7fdf01759a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:21:57.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:21:57.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:21:57.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:21:57.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:21:57.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:21:57.842 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:21:57.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:57.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:21:57.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:57.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:57.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:58.241 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:58.237+0000 7fdf01759a40 -1 Falling back to public interface 2026-02-21T01:21:58.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:21:58.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:21:58.955 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:21:58.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:21:58.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:21:58.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:21:59.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:21:59.186 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:21:59.181+0000 7fdf01759a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:22:00.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:00.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:00.067 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:22:00.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:22:00.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:00.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:00.284 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/5993523,v1:127.0.0.1:6801/5993523] [v2:127.0.0.1:6802/5993523,v1:127.0.0.1:6803/5993523] exists,up a0e58c06-dc32-419a-9bcd-7f82e1ca72f9 2026-02-21T01:22:00.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:00.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:00.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:00.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1558: test_is_clean: create_rbd_pool 2026-02-21T01:22:00.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:22:00.490 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:22:00.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:22:00.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:22:00.730 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:22:00.740 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:22:01.741 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:22:02.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: test_is_clean: wait_for_clean 2026-02-21T01:22:02.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:22:02.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:22:02.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:22:02.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:22:02.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:22:02.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:22:02.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:22:02.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:22:02.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:22:02.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:22:02.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:22:02.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:22:02.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:22:02.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:22:02.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:22:02.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:22:02.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:22:02.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:22:02.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:22:02.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:22:02.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:22:02.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:22:02.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:22:02.398 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:22:02.398 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:22:02.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:22:02.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:22:02.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:22:02.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:22:02.401 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:22:02.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:22:02.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:02.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:22:02.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:03.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:22:03.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:03.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:22:03.834 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:22:03.834 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:03.834 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:04.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:22:04.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:22:04.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:22:04.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:22:04.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:04.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:04.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:22:04.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:22:04.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:22:04.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1560: test_is_clean: is_clean 2026-02-21T01:22:04.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: get_num_pgs 2026-02-21T01:22:04.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:04.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:04.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: num_pgs=4 2026-02-21T01:22:04.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1547: is_clean: test 4 '!=' 0 2026-02-21T01:22:04.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: get_num_active_clean 2026-02-21T01:22:04.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:22:04.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:22:04.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:22:04.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:22:04.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: test 4 = 4 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: teardown td/ceph-helpers 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:05.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:05.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:05.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:05.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:05.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:05.173 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:05.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:05.174 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:05.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:05.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:05.175 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:05.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:05.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:05.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:05.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:05.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:05.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:05.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:05.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:05.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:05.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_kill_daemon td/ceph-helpers 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: test_kill_daemon: local dir=td/ceph-helpers 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:267: test_kill_daemon: setup td/ceph-helpers 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:05.188 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:05.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:05.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:05.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:05.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:05.190 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:05.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:05.191 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:05.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:05.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:05.193 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:05.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:05.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:05.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:05.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:05.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:05.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:05.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:05.198 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:05.198 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.198 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:05.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:05.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:05.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:22:05.200 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:22:05.200 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:22:05.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:22:05.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:22:05.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:22:05.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:05.224 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:22:05.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:05.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:05.225 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:05.225 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.225 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:05.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:05.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:22:05.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:22:05.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:05.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:05.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:22:05.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:22:05.254 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:05.254 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:05.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:22:05.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:22:05.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:05.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:05.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:22:05.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.311 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:05.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:05.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:22:05.364 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: test_kill_daemon: run_mgr td/ceph-helpers x 2026-02-21T01:22:05.364 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:22:05.364 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:22:05.365 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:22:05.365 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:22:05.365 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:22:05.365 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.477 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:05.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:05.479 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:05.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:270: test_kill_daemon: run_osd td/ceph-helpers 0 2026-02-21T01:22:05.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:22:05.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:22:05.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:05.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:22:05.505 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:22:05.506 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:22:05.506 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=27054143-ad82-4de5-b4af-eafbc1dc4491 2026-02-21T01:22:05.507 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 27054143-ad82-4de5-b4af-eafbc1dc4491' 2026-02-21T01:22:05.507 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 27054143-ad82-4de5-b4af-eafbc1dc4491 2026-02-21T01:22:05.507 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:22:05.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC9CJlpzjm3HhAAOcW67NcYHNaRWjR0yjuk4Q== 2026-02-21T01:22:05.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC9CJlpzjm3HhAAOcW67NcYHNaRWjR0yjuk4Q=="}' 2026-02-21T01:22:05.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 27054143-ad82-4de5-b4af-eafbc1dc4491 -i td/ceph-helpers/0/new.json 2026-02-21T01:22:05.629 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:05.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:22:05.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9CJlpzjm3HhAAOcW67NcYHNaRWjR0yjuk4Q== --osd-uuid 27054143-ad82-4de5-b4af-eafbc1dc4491 2026-02-21T01:22:05.655 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:05.649+0000 7f3341a00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:05.656 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:05.653+0000 7f3341a00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:05.657 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:05.653+0000 7f3341a00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:05.658 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:05.653+0000 7f3341a00a40 -1 bdev(0x563006898c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:22:05.658 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:05.653+0000 7f3341a00a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:22:06.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:22:06.414 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:22:06.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:22:06.416 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:22:06.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:22:06.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:22:06.536 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:22:06.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:22:06.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:22:06.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:22:06.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:22:06.561 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:06.561+0000 7f37b14e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:06.563 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:06.561+0000 7f37b14e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:06.564 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:06.565+0000 7f37b14e1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:06.643 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:22:06.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:22:06.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:22:06.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:22:06.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:22:06.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:06.645 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:06.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:22:06.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:06.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:06.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:07.042 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:07.033+0000 7f37b14e1a40 -1 Falling back to public interface 2026-02-21T01:22:07.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:07.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:07.752 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:22:07.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:22:07.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:07.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:07.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:08.005 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:08.004+0000 7f37b14e1a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:22:08.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:08.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:08.864 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:22:08.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:22:08.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:08.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:09.075 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4156157159,v1:127.0.0.1:6801/4156157159] [v2:127.0.0.1:6802/4156157159,v1:127.0.0.1:6803/4156157159] exists,up 27054143-ad82-4de5-b4af-eafbc1dc4491 2026-02-21T01:22:09.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:09.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:09.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:09.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:272: test_kill_daemon: name_prefix=osd 2026-02-21T01:22:09.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: find td/ceph-helpers 2026-02-21T01:22:09.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: grep 'osd[^/]*\.pid' 2026-02-21T01:22:09.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-21T01:22:09.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:280: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2026-02-21T01:22:09.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-21T01:22:09.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=102419 2026-02-21T01:22:09.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=0 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local delays=1 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 102419 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 1 2026-02-21T01:22:10.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 102419 2026-02-21T01:22:10.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:10.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:10.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0 2026-02-21T01:22:10.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 1 2026-02-21T01:22:10.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:284: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2026-02-21T01:22:10.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=102419 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 102419 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:10.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-21T01:22:10.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 102419 2026-02-21T01:22:10.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:10.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:10.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2026-02-21T01:22:10.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 102419 2026-02-21T01:22:10.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-21T01:22:10.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-21T01:22:10.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-21T01:22:10.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:287: test_kill_daemon: name_prefix=mgr 2026-02-21T01:22:10.388 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: find td/ceph-helpers 2026-02-21T01:22:10.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: grep 'mgr[^/]*\.pid' 2026-02-21T01:22:10.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-21T01:22:10.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:292: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2026-02-21T01:22:10.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2026-02-21T01:22:10.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=101563 2026-02-21T01:22:10.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-21T01:22:10.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-21T01:22:10.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T01:22:10.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 101563 2026-02-21T01:22:10.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:10.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:10.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-21T01:22:10.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 101563 2026-02-21T01:22:10.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-21T01:22:10.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-21T01:22:10.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-21T01:22:10.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:295: test_kill_daemon: name_prefix=mon 2026-02-21T01:22:10.495 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: find td/ceph-helpers 2026-02-21T01:22:10.495 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: grep 'mon[^/]*\.pid' 2026-02-21T01:22:10.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-21T01:22:10.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:300: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2026-02-21T01:22:10.497 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mon.a.pid 2026-02-21T01:22:10.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=101478 2026-02-21T01:22:10.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-21T01:22:10.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-21T01:22:10.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T01:22:10.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 101478 2026-02-21T01:22:10.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:10.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:10.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-21T01:22:10.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 101478 2026-02-21T01:22:10.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T01:22:10.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T01:22:10.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2026-02-21T01:22:10.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T01:22:10.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 101478 2026-02-21T01:22:10.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-21T01:22:10.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-21T01:22:10.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-21T01:22:10.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: timeout 5 ceph status 2026-02-21T01:22:15.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: teardown td/ceph-helpers 2026-02-21T01:22:15.808 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:15.808 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:15.808 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:15.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:15.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:15.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:15.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:15.809 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:15.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:15.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:15.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:15.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:15.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:15.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:15.817 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:15.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:15.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:15.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:15.819 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:15.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:15.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:15.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:15.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:15.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:15.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:15.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:15.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_kill_daemons td/ceph-helpers 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:357: test_kill_daemons: local dir=td/ceph-helpers 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:358: test_kill_daemons: setup td/ceph-helpers 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:15.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:15.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:15.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:15.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:15.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:15.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:15.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:15.829 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:15.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:15.830 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:15.832 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:15.832 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:15.832 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:15.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:15.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:15.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:15.833 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:15.834 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:15.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:15.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:15.837 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:15.837 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:15.837 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:15.837 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:15.838 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:15.838 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:15.838 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:22:15.840 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:22:15.840 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:15.840 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:15.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:22:15.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:22:15.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:359: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:22:15.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:15.865 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:22:15.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:15.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:15.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:15.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:15.866 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:15.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:15.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:15.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:22:15.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:22:15.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:15.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:15.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:22:15.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:15.898 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:22:15.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:22:15.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:15.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:15.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:22:15.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:15.957 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:22:16.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:360: test_kill_daemons: run_mgr td/ceph-helpers x 2026-02-21T01:22:16.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:22:16.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:22:16.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:22:16.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:22:16.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:22:16.012 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:22:16.127 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:22:16.127 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:16.127 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:16.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:16.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:16.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:16.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:16.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:16.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:361: test_kill_daemons: run_osd td/ceph-helpers 0 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:16.150 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:16.151 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:22:16.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:22:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:22:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:22:16.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:22:16.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:22:16.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=94f5e34f-cd5b-4d38-bbb7-8ea86b0ef104 2026-02-21T01:22:16.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 94f5e34f-cd5b-4d38-bbb7-8ea86b0ef104' 2026-02-21T01:22:16.154 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 94f5e34f-cd5b-4d38-bbb7-8ea86b0ef104 2026-02-21T01:22:16.154 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:22:16.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDICJlp4o3HCRAAEioKPGe/M80niidBGNwBMw== 2026-02-21T01:22:16.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDICJlp4o3HCRAAEioKPGe/M80niidBGNwBMw=="}' 2026-02-21T01:22:16.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 94f5e34f-cd5b-4d38-bbb7-8ea86b0ef104 -i td/ceph-helpers/0/new.json 2026-02-21T01:22:16.276 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:16.284 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:22:16.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDICJlp4o3HCRAAEioKPGe/M80niidBGNwBMw== --osd-uuid 94f5e34f-cd5b-4d38-bbb7-8ea86b0ef104 2026-02-21T01:22:16.301 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:16.300+0000 7fd157125a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:16.303 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:16.304+0000 7fd157125a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:16.304 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:16.304+0000 7fd157125a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:16.305 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:16.304+0000 7fd157125a40 -1 bdev(0x5569b88a2c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:22:16.305 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:16.304+0000 7fd157125a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:22:16.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:22:16.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:22:16.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:22:16.956 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:22:16.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:22:17.069 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:22:17.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:22:17.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:22:17.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:22:17.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:22:17.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:22:17.085 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:17.084+0000 7f78a1171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:17.086 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:17.084+0000 7f78a1171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:17.087 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:17.084+0000 7f78a1171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:17.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:22:17.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:22:17.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:22:17.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:22:17.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:22:17.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:17.184 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:17.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:22:17.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:17.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:17.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:17.529 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:17.528+0000 7f78a1171a40 -1 Falling back to public interface 2026-02-21T01:22:18.172 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:18.172+0000 7f78a1171a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:22:18.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:18.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:18.292 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:22:18.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:22:18.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:18.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:18.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:19.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:19.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:19.442 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:22:19.443 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:22:19.443 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:19.443 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:19.657 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4294728657,v1:127.0.0.1:6801/4294728657] [v2:127.0.0.1:6802/4294728657,v1:127.0.0.1:6803/4294728657] exists,up 94f5e34f-cd5b-4d38-bbb7-8ea86b0ef104 2026-02-21T01:22:19.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:19.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:19.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:19.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2026-02-21T01:22:19.659 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:19.659 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:19.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:19.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:19.659 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:20.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 1 2026-02-21T01:22:20.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:372: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2026-02-21T01:22:20.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:20.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:20.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:20.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:20.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:20.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:20.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:376: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2026-02-21T01:22:20.978 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:20.978 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:20.978 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:380: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:21.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:21.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:21.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: timeout 5 ceph status 2026-02-21T01:22:26.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:382: test_kill_daemons: teardown td/ceph-helpers 2026-02-21T01:22:26.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:26.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:26.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:26.197 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:26.197 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:26.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:26.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:26.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:26.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:26.203 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:26.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:26.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:26.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:26.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:26.206 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:26.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:26.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:26.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:26.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:26.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:26.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:26.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:26.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:26.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:26.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:26.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_objectstore_tool td/ceph-helpers 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: test_objectstore_tool: local dir=td/ceph-helpers 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: test_objectstore_tool: setup td/ceph-helpers 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:26.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:26.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:26.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:26.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:26.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:26.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:26.221 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:26.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:26.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:26.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:26.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:26.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:26.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:26.224 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:26.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:26.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:26.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:26.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:26.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:26.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:26.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:26.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:22:26.231 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:22:26.231 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.231 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.231 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:22:26.234 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:22:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:22:26.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:26.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:26.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:22:26.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:22:26.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:26.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:26.287 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:26.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:22:26.345 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:26.346 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:26.347 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:26.347 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.347 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.348 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:26.348 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:26.348 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:22:26.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: test_objectstore_tool: local osd=0 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: test_objectstore_tool: run_mgr td/ceph-helpers x 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:22:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:22:26.516 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:26.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:26.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:26.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: test_objectstore_tool: run_osd td/ceph-helpers 0 2026-02-21T01:22:26.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:22:26.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:22:26.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:22:26.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:22:26.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:22:26.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:22:26.542 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:22:26.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8c782cdf-def0-46ce-8735-cd25516230f0 2026-02-21T01:22:26.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8c782cdf-def0-46ce-8735-cd25516230f0' 2026-02-21T01:22:26.542 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 8c782cdf-def0-46ce-8735-cd25516230f0 2026-02-21T01:22:26.543 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:22:26.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDSCJlp+eH2IBAAbkN4KueQ4ZwCeEysoUMlhQ== 2026-02-21T01:22:26.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDSCJlp+eH2IBAAbkN4KueQ4ZwCeEysoUMlhQ=="}' 2026-02-21T01:22:26.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8c782cdf-def0-46ce-8735-cd25516230f0 -i td/ceph-helpers/0/new.json 2026-02-21T01:22:26.662 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:26.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:22:26.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSCJlp+eH2IBAAbkN4KueQ4ZwCeEysoUMlhQ== --osd-uuid 8c782cdf-def0-46ce-8735-cd25516230f0 2026-02-21T01:22:26.686 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:26.684+0000 7f4631856a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:26.688 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:26.688+0000 7f4631856a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:26.689 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:26.688+0000 7f4631856a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:26.689 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:26.688+0000 7f4631856a40 -1 bdev(0x559db508ec00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:22:26.689 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:26.688+0000 7f4631856a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:22:27.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:22:27.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:22:27.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:22:27.452 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:22:27.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:22:27.577 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:22:27.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:22:27.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:22:27.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:22:27.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:22:27.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:22:27.600 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:27.600+0000 7f5390320a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:27.602 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:27.600+0000 7f5390320a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:27.603 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:27.600+0000 7f5390320a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:27.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:22:27.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:22:27.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:22:27.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:22:27.691 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:22:27.691 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:27.691 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:27.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:22:27.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:27.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:27.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:28.017 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:28.016+0000 7f5390320a40 -1 Falling back to public interface 2026-02-21T01:22:28.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:28.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:28.804 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:22:28.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:22:28.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:28.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:28.984 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:28.984+0000 7f5390320a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:22:29.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:30.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:30.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:30.022 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:22:30.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:22:30.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:30.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:30.239 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2850735635,v1:127.0.0.1:6801/2850735635] [v2:127.0.0.1:6802/2850735635,v1:127.0.0.1:6803/2850735635] exists,up 8c782cdf-def0-46ce-8735-cd25516230f0 2026-02-21T01:22:30.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:30.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:30.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:30.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: test_objectstore_tool: create_rbd_pool 2026-02-21T01:22:30.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:22:30.448 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:22:30.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:22:30.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:22:30.757 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:22:30.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:22:31.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:22:32.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: test_objectstore_tool: wait_for_clean 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:22:32.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:22:32.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:22:32.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:22:32.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:22:32.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:22:32.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:22:32.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:22:32.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:22:32.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:22:32.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:22:32.375 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:22:32.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:22:32.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:22:32.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:22:32.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:22:32.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:22:32.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:22:32.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:22:32.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:22:32.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:22:32.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:22:32.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:22:32.455 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:22:32.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:32.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:22:32.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:33.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:22:33.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:33.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:22:33.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:22:33.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:33.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:34.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:22:34.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:22:34.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:22:34.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:22:34.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:22:34.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:22:34.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:22:34.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:22:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:22:34.388 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:22:34.388 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:34.388 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:34.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:22:34.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:22:34.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:22:34.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T01:22:34.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: diff - /etc/group 2026-02-21T01:22:34.686 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:34.686 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:34.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:34.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:34.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:34.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:34.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2026-02-21T01:22:34.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-21T01:22:34.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-21T01:22:34.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-21T01:22:34.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-21T01:22:34.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:34.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2026-02-21T01:22:35.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2026-02-21T01:22:35.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:22:35.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:22:35.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:22:35.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:22:35.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:35.157 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:22:35.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:35.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:22:35.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:22:35.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:22:35.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:22:35.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:22:35.161 INFO:tasks.workunit.client.0.trial106.stderr:start osd.0 2026-02-21T01:22:35.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:22:35.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T01:22:35.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:22:35.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:22:35.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:22:35.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:22:35.175 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:35.172+0000 7f2cdd975a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:35.176 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:35.176+0000 7f2cdd975a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:35.177 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:35.176+0000 7f2cdd975a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:35.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:22:35.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:22:35.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:22:35.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:22:35.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:22:35.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:35.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:22:35.374 INFO:tasks.workunit.client.0.trial106.stderr:0 2026-02-21T01:22:35.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:35.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:35.557 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:35.556+0000 7f2cdd975a40 -1 Falling back to public interface 2026-02-21T01:22:35.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:36.187 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:36.188+0000 7f2cdd975a40 -1 osd.0 12 log_to_monitors true 2026-02-21T01:22:36.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:36.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:36.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:22:36.583 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:22:36.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:36.583 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:36.802 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 up in weight 1 up_from 15 up_thru 15 down_at 13 last_clean_interval [5,12) [v2:127.0.0.1:6800/120225198,v1:127.0.0.1:6801/120225198] [v2:127.0.0.1:6802/120225198,v1:127.0.0.1:6803/120225198] exists,up 8c782cdf-def0-46ce-8735-cd25516230f0 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:22:36.803 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:22:36.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:22:36.804 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:22:36.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:22:36.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:22:36.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:22:36.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:22:36.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:22:36.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:22:37.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:22:37.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:22:37.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:22:37.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:22:37.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-21T01:22:37.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-21T01:22:37.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-64424509443' 2026-02-21T01:22:37.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:22:37.183 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-64424509443 2026-02-21T01:22:37.183 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:22:37.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:22:37.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-64424509443 2026-02-21T01:22:37.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:22:37.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-21T01:22:37.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 64424509443' 2026-02-21T01:22:37.186 INFO:tasks.workunit.client.0.trial106.stderr:waiting osd.0 seq 64424509443 2026-02-21T01:22:37.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:37.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 64424509443 2026-02-21T01:22:37.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:38.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:22:38.403 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:38.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 64424509443 2026-02-21T01:22:38.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:39.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:22:39.616 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:39.832 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-21T01:22:39.832 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:22:39.833 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:39.833 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:40.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:22:40.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:22:40.119 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:22:40.119 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:22:40.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:22:40.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:22:40.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:22:40.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:22:40.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:22:40.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:22:40.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:40.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-21T01:22:40.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:40.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:40.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:40.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2026-02-21T01:22:40.993 INFO:tasks.workunit.client.0.trial106.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2026-02-21T01:22:41.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: return 1 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: return 1 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: teardown td/ceph-helpers 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:41.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:41.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:41.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:41.188 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:41.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:41.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:41.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:41.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:41.190 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:41.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:41.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:41.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:41.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:41.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:41.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:41.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:41.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:41.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_pg_schedule_scrub td/ceph-helpers 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: test_pg_schedule_scrub: local dir=td/ceph-helpers 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: test_pg_schedule_scrub: setup td/ceph-helpers 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:41.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:41.203 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:41.203 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:41.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:41.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:41.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:41.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:41.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:41.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:41.207 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:41.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:41.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:41.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:41.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:41.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:41.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:41.210 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:41.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:41.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:41.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:41.213 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:41.213 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.213 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:41.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:41.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:41.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:22:41.216 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:22:41.216 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.216 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:22:41.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:22:41.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:22:41.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:22:41.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:41.240 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:22:41.240 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:41.240 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:41.241 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:41.241 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.241 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.241 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:41.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:41.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:22:41.270 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:41.271 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:41.272 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:41.272 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.272 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.272 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:41.272 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:41.272 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:22:41.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:22:41.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:41.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:41.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:22:41.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:41.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:22:41.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_pg_schedule_scrub: run_mgr td/ceph-helpers x --mgr_stats_period=1 2026-02-21T01:22:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:22:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:22:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:22:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:22:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:22:41.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:22:41.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:22:41.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:41.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:41.493 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:41.493 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.493 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.493 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:41.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:41.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:41.515 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:22:41.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:22:41.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:22:41.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7007b091-fd54-4cef-989e-35bb9bd87298 2026-02-21T01:22:41.519 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 7007b091-fd54-4cef-989e-35bb9bd87298 2026-02-21T01:22:41.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7007b091-fd54-4cef-989e-35bb9bd87298' 2026-02-21T01:22:41.519 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:22:41.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhCJlpA+qqHxAAne6Lh2naUPcA9CPqQaU5oQ== 2026-02-21T01:22:41.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhCJlpA+qqHxAAne6Lh2naUPcA9CPqQaU5oQ=="}' 2026-02-21T01:22:41.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7007b091-fd54-4cef-989e-35bb9bd87298 -i td/ceph-helpers/0/new.json 2026-02-21T01:22:41.645 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:41.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:22:41.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhCJlpA+qqHxAAne6Lh2naUPcA9CPqQaU5oQ== --osd-uuid 7007b091-fd54-4cef-989e-35bb9bd87298 2026-02-21T01:22:41.670 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:41.668+0000 7f445d2faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:41.672 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:41.672+0000 7f445d2faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:41.672 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:41.672+0000 7f445d2faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:41.673 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:41.672+0000 7f445d2faa40 -1 bdev(0x55b463888c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:22:41.673 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:41.672+0000 7f445d2faa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:22:42.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:22:42.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:22:42.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:22:42.428 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:22:42.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:22:42.549 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:22:42.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:22:42.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:22:42.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:22:42.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:22:42.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:22:42.573 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:42.572+0000 7ffab706da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:42.575 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:42.576+0000 7ffab706da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:42.577 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:42.576+0000 7ffab706da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:42.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:22:42.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:22:42.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:22:42.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:22:42.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:22:42.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:42.657 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:42.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:22:42.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:42.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:42.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:43.077 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:43.076+0000 7ffab706da40 -1 Falling back to public interface 2026-02-21T01:22:43.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:43.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:43.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:22:43.772 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:22:43.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:43.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:43.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:44.059 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:44.056+0000 7ffab706da40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:22:44.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:44.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:44.877 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:22:44.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:22:44.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:44.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:45.093 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4258941661,v1:127.0.0.1:6801/4258941661] [v2:127.0.0.1:6802/4258941661,v1:127.0.0.1:6803/4258941661] exists,up 7007b091-fd54-4cef-989e-35bb9bd87298 2026-02-21T01:22:45.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:45.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:45.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:45.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_pg_schedule_scrub: create_rbd_pool 2026-02-21T01:22:45.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:22:45.291 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:22:45.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:22:45.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:22:45.535 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:22:45.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:22:46.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:22:46.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_pg_schedule_scrub: wait_for_clean 2026-02-21T01:22:46.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:22:46.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:22:46.824 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:22:46.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:22:46.825 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:22:46.825 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:22:46.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:22:46.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:22:46.825 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:22:46.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:22:46.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:22:46.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:22:46.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:22:46.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:22:46.926 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:22:47.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:22:47.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:22:47.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:22:47.143 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:22:47.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-21T01:22:47.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-21T01:22:47.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T01:22:47.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:22:47.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-21T01:22:47.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:22:47.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:22:47.224 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-21T01:22:47.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:22:47.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-21T01:22:47.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T01:22:47.226 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836484 2026-02-21T01:22:47.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:47.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836484 2026-02-21T01:22:47.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:48.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:22:48.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:48.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836484 2026-02-21T01:22:48.666 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:22:48.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:48.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:22:48.944 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:22:49.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:22:49.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:22:49.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:22:49.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:22:49.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: pg_schedule_scrub: local pgid=1.0 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: pg_schedule_scrub: wait_for_pg_clean 1.0 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:22:49.417 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:22:49.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:22:49.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:22:49.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-21T01:22:49.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-21T01:22:49.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-21T01:22:49.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-21T01:22:49.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:22:49.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:22:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:22:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:22:49.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:22:49.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:22:49.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-21T01:22:49.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-21T01:22:49.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-21T01:22:49.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:22:49.969 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-21T01:22:49.969 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:22:49.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:22:49.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-21T01:22:49.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:22:49.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-21T01:22:49.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-21T01:22:49.972 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836489 2026-02-21T01:22:49.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:50.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-21T01:22:50.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:51.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:22:51.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:51.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-21T01:22:51.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:22:52.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:22:52.411 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836489 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 0 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:22:52.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:22:52.616 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:22:52.616 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:22:52.696 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: pg_schedule_scrub: get_last_scrub_stamp 1.0 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:22:52.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:22:52.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: pg_schedule_scrub: local last_scrub=2026-02-21T01:22:45.483701+0000 2026-02-21T01:22:52.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: pg_schedule_scrub: ceph tell 1.0 schedule-scrub 2026-02-21T01:22:52.979 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:22:52.979 INFO:tasks.workunit.client.0.trial106.stdout: "deep": false, 2026-02-21T01:22:52.979 INFO:tasks.workunit.client.0.trial106.stdout: "must": false, 2026-02-21T01:22:52.979 INFO:tasks.workunit.client.0.trial106.stdout: "stamp": "2026-02-18T13:22:52.981170+0000" 2026-02-21T01:22:52.980 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: pg_schedule_scrub: wait_for_scrub 1.0 2026-02-21T01:22:45.483701+0000 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-21T01:22:45.483701+0000 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:22:52.988 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:22:52.989 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:22:52.989 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:22:53.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:22:45.483701+0000 '>' 2026-02-21T01:22:45.483701+0000 2026-02-21T01:22:53.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:22:54.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:22:54.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:22:54.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:22:54.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:22:54.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:22:54.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:22:54.205 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:22:53.112313+0000 '>' 2026-02-21T01:22:45.483701+0000 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_pg_schedule_scrub: teardown td/ceph-helpers 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:54.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:54.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:54.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:54.526 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:54.527 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:54.527 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:54.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:54.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:54.529 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:54.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:54.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:54.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:54.530 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:54.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:54.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:54.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:54.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_pg_scrub td/ceph-helpers 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_scrub: local dir=td/ceph-helpers 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: test_pg_scrub: setup td/ceph-helpers 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:22:54.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:22:54.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:22:54.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:22:54.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:22:54.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:22:54.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:22:54.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:22:54.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:22:54.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:22:54.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:54.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:22:54.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:22:54.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:22:54.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:22:54.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:22:54.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:22:54.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:22:54.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:22:54.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:22:54.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:22:54.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:22:54.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:22:54.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:22:54.555 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:22:54.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1944: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:22:54.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:54.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:22:54.580 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:54.580 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:54.580 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:54.580 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.580 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.580 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:54.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:22:54.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:22:54.611 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:22:54.611 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:54.611 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:54.611 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:22:54.611 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:22:54.612 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:54.612 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:54.612 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:54.612 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:54.613 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.613 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.613 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:54.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:54.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:22:54.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:22:54.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:22:54.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:22:54.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:22:54.670 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:22:54.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: test_pg_scrub: run_mgr td/ceph-helpers x 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:22:54.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:22:54.839 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:22:54.839 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:54.839 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:54.840 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:54.840 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.840 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.840 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:54.841 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:54.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: test_pg_scrub: run_osd td/ceph-helpers 0 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:22:54.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:54.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:22:54.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:22:54.863 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:22:54.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a40b190-6837-4acb-90dd-7b79d6cf1494 2026-02-21T01:22:54.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3a40b190-6837-4acb-90dd-7b79d6cf1494' 2026-02-21T01:22:54.864 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 3a40b190-6837-4acb-90dd-7b79d6cf1494 2026-02-21T01:22:54.864 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:22:54.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDuCJlpdkFJNBAAsJOTGn2MDERq7cKD82BZLA== 2026-02-21T01:22:54.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDuCJlpdkFJNBAAsJOTGn2MDERq7cKD82BZLA=="}' 2026-02-21T01:22:54.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a40b190-6837-4acb-90dd-7b79d6cf1494 -i td/ceph-helpers/0/new.json 2026-02-21T01:22:54.989 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:54.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:22:54.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuCJlpdkFJNBAAsJOTGn2MDERq7cKD82BZLA== --osd-uuid 3a40b190-6837-4acb-90dd-7b79d6cf1494 2026-02-21T01:22:55.014 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.012+0000 7f4115919a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:55.016 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.016+0000 7f4115919a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:55.017 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.016+0000 7f4115919a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:55.018 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.016+0000 7f4115919a40 -1 bdev(0x564345060c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:22:55.018 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.016+0000 7f4115919a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:22:55.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:22:55.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:22:55.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:22:55.688 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:22:55.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:22:55.803 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:22:55.803 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:22:55.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:22:55.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:22:55.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:22:55.804 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:22:55.819 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.820+0000 7f28de596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:55.820 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.820+0000 7f28de596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:55.821 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:55.820+0000 7f28de596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:22:55.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:22:55.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:22:55.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:22:55.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:22:55.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:22:55.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:55.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:22:55.922 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:22:55.923 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:55.923 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:56.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:56.213 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:56.212+0000 7f28de596a40 -1 Falling back to public interface 2026-02-21T01:22:56.913 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:22:56.912+0000 7f28de596a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:22:57.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:57.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:57.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:22:57.029 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:22:57.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:57.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:57.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:22:58.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:22:58.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:22:58.188 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:22:58.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:22:58.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:22:58.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:22:58.407 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3580950639,v1:127.0.0.1:6801/3580950639] [v2:127.0.0.1:6802/3580950639,v1:127.0.0.1:6803/3580950639] exists,up 3a40b190-6837-4acb-90dd-7b79d6cf1494 2026-02-21T01:22:58.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:22:58.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:22:58.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:22:58.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_scrub: create_rbd_pool 2026-02-21T01:22:58.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:22:58.615 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:22:58.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:22:58.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:22:58.865 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:22:58.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:22:59.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:23:00.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_pg_scrub: wait_for_clean 2026-02-21T01:23:00.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:23:00.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:23:00.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:23:00.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:23:00.156 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:23:00.157 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:23:00.157 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:23:00.157 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:23:00.157 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:23:00.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:23:00.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:23:00.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:23:00.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:23:00.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:23:00.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:23:00.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:23:00.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:23:00.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:23:00.469 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:23:00.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:23:00.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:23:00.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:23:00.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:23:00.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:23:00.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:23:00.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:23:00.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:23:00.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:23:00.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:23:00.551 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:23:00.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:23:00.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:23:00.764 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:23:00.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:23:01.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:23:01.767 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:23:01.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:23:01.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:23:01.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:23:01.975 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:23:02.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:23:02.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:23:02.454 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:23:02.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:23:02.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_scrub: pg_scrub 1.0 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-21T01:23:02.725 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-21T01:23:02.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-21T01:23:02.726 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:23:02.726 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:23:02.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:23:02.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:23:02.727 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:23:02.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-21T01:23:02.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-21T01:23:02.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-21T01:23:02.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-21T01:23:02.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:23:02.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:23:03.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:23:03.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:23:03.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:23:03.212 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:23:03.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-21T01:23:03.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-21T01:23:03.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-21T01:23:03.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:23:03.290 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-21T01:23:03.290 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:23:03.291 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:23:03.291 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-21T01:23:03.291 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:23:03.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-21T01:23:03.293 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836485 2026-02-21T01:23:03.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-21T01:23:03.293 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:23:03.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-21T01:23:03.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:23:04.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:23:04.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:23:04.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-21T01:23:04.722 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:23:05.723 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:23:05.724 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:23:05.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-21T01:23:05.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:23:05.938 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-21T01:23:05.939 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 0 2026-02-21T01:23:05.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:23:05.939 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:23:05.940 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:23:05.940 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:23:05.940 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:23:06.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-21T01:23:06.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:23:06.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-21T01:23:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-21T01:23:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-21T01:23:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:23:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:23:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:23:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:23:06.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:06.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-21T01:23:06.434 INFO:tasks.workunit.client.0.trial106.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-21T01:23:06.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:06.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-21T01:23:06.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:23:06.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:23:06.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:22:58.815012+0000 '>' 2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:06.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:23:07.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:23:07.662 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:23:07.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:23:07.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:23:07.662 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:23:07.663 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:23:07.663 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:23:07.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:22:58.815012+0000 '>' 2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:07.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:23:08.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:23:08.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:23:08.878 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:23:08.878 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:23:08.878 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:23:08.879 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:23:08.879 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:23:09.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:22:58.815012+0000 '>' 2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:09.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:23:10.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:23:10.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:23:10.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:23:10.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:23:10.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:23:10.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:23:10.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:23:10.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:23:06.707996+0000 '>' 2026-02-21T01:22:58.815012+0000 2026-02-21T01:23:10.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-21T01:23:10.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-21T01:23:10.303 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:23:10.303 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:23:10.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:23:10.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:23:10.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: TIMEOUT=2 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: pg_scrub 1.0 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-21T01:23:10.411 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-21T01:23:10.412 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:23:10.412 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:23:10.412 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:23:10.412 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:23:10.412 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:23:10.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-21T01:23:10.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-21T01:23:10.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-21T01:23:10.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-21T01:23:10.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=2 2026-02-21T01:23:10.666 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:23:10.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:23:10.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:23:10.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:23:10.880 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:37:59.652 INFO:tasks.workunit.client.0.trial106.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-21T01:37:59.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-21T01:37:59.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-21T01:37:59.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2261: flush_pg_stats: continue 2026-02-21T01:37:59.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:37:59.663 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-21T01:37:59.663 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 0 2026-02-21T01:37:59.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:37:59.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:37:59.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:37:59.664 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:37:59.664 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:37:59.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:37:59.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:37:59.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 0 >= 31 )) 2026-02-21T01:37:59.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 1 2026-02-21T01:38:00.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:00.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:00.744 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 1' 2026-02-21T01:38:00.744 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 1 2026-02-21T01:38:00.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:00.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:00.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:00.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:00.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:00.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:00.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:00.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 1 >= 31 )) 2026-02-21T01:38:00.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 2 2026-02-21T01:38:02.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:02.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:02.829 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 2 2026-02-21T01:38:02.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 2' 2026-02-21T01:38:02.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:02.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:02.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:02.831 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:02.831 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:02.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:02.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:02.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 2 >= 31 )) 2026-02-21T01:38:02.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:05.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:05.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 3 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 3' 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:05.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:05.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:05.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:05.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 3 >= 31 )) 2026-02-21T01:38:05.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:08.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:08.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:08.996 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 4 2026-02-21T01:38:08.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 4' 2026-02-21T01:38:08.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:08.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:08.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:08.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:08.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:09.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:09.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:09.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 4 >= 31 )) 2026-02-21T01:38:09.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:12.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:12.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:12.081 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 5 2026-02-21T01:38:12.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 5' 2026-02-21T01:38:12.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:12.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:12.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:12.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:12.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:12.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:12.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:12.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 5 >= 31 )) 2026-02-21T01:38:12.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:15.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:15.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:15.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 6' 2026-02-21T01:38:15.167 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 6 2026-02-21T01:38:15.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:15.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:15.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:15.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:15.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:15.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:15.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:15.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 6 >= 31 )) 2026-02-21T01:38:15.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:18.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:18.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:18.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 7' 2026-02-21T01:38:18.250 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 7 2026-02-21T01:38:18.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:18.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:18.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:18.251 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:18.251 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:18.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:18.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:18.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 7 >= 31 )) 2026-02-21T01:38:18.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:21.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:21.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:21.335 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 8 2026-02-21T01:38:21.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 8' 2026-02-21T01:38:21.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:21.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:21.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:21.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:21.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:21.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:21.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:21.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 8 >= 31 )) 2026-02-21T01:38:21.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 9 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 9' 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:24.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:24.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:24.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:24.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:24.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:24.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 9 >= 31 )) 2026-02-21T01:38:24.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:27.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:27.502 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:27.502 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 10 2026-02-21T01:38:27.503 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 10' 2026-02-21T01:38:27.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:27.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:27.504 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:27.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:27.504 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:27.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:27.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:27.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 10 >= 31 )) 2026-02-21T01:38:27.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:30.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:30.588 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:30.588 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 11 2026-02-21T01:38:30.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 11' 2026-02-21T01:38:30.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:30.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:30.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:30.589 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:30.589 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:30.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:30.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:30.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 11 >= 31 )) 2026-02-21T01:38:30.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:33.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:33.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:33.674 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 12 2026-02-21T01:38:33.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 12' 2026-02-21T01:38:33.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:33.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:33.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:33.675 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:33.675 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:33.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:33.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:33.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 12 >= 31 )) 2026-02-21T01:38:33.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:36.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:36.757 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:36.757 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 13 2026-02-21T01:38:36.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 13' 2026-02-21T01:38:36.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:36.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:36.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:36.758 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:36.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:36.839 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:36.839 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:36.839 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 13 >= 31 )) 2026-02-21T01:38:36.839 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:39.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:39.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:39.841 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 14 2026-02-21T01:38:39.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 14' 2026-02-21T01:38:39.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:39.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:39.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:39.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:39.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:39.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:39.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:39.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 14 >= 31 )) 2026-02-21T01:38:39.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:42.923 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:42.924 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:42.924 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 15 2026-02-21T01:38:42.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 15' 2026-02-21T01:38:42.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:42.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:42.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:42.925 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:42.925 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:43.004 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:43.004 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:43.004 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 15 >= 31 )) 2026-02-21T01:38:43.004 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 16 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 16' 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:46.006 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:46.007 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:46.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:46.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:46.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 16 >= 31 )) 2026-02-21T01:38:46.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 17' 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 17 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:49.094 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:49.095 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:49.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:49.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:49.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 17 >= 31 )) 2026-02-21T01:38:49.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:52.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:52.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:52.182 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 18 2026-02-21T01:38:52.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 18' 2026-02-21T01:38:52.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:52.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:52.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:52.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:52.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:52.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:52.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:52.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 18 >= 31 )) 2026-02-21T01:38:52.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:55.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:55.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:55.267 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 19 2026-02-21T01:38:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 19' 2026-02-21T01:38:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:55.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:55.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:55.270 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:55.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:55.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:55.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 19 >= 31 )) 2026-02-21T01:38:55.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:38:58.353 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:38:58.353 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:38:58.353 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 20' 2026-02-21T01:38:58.354 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 20 2026-02-21T01:38:58.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:38:58.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:38:58.354 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:38:58.354 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:38:58.354 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:38:58.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:38:58.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:38:58.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 20 >= 31 )) 2026-02-21T01:38:58.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:01.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 21 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 21' 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:01.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:01.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:01.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:01.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 21 >= 31 )) 2026-02-21T01:39:01.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:04.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:04.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:04.524 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 22 2026-02-21T01:39:04.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 22' 2026-02-21T01:39:04.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:04.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:04.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:04.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:04.525 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:04.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:04.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:04.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 22 >= 31 )) 2026-02-21T01:39:04.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:07.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 23 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 23' 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:07.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:07.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:07.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:07.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 23 >= 31 )) 2026-02-21T01:39:07.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:10.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:10.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:10.689 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 24 2026-02-21T01:39:10.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 24' 2026-02-21T01:39:10.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:10.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:10.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:10.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:10.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:10.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:10.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:10.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 24 >= 31 )) 2026-02-21T01:39:10.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:13.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:13.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 25 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 25' 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:13.774 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:13.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:13.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:13.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 25 >= 31 )) 2026-02-21T01:39:13.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 26' 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 26 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:16.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:16.862 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:16.862 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:16.942 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:16.942 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:16.942 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 26 >= 31 )) 2026-02-21T01:39:16.942 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:19.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:19.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:19.944 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 27 2026-02-21T01:39:19.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 27' 2026-02-21T01:39:19.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:19.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:19.946 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:19.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:19.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:20.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:20.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:20.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 27 >= 31 )) 2026-02-21T01:39:20.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 28 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 28' 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:23.026 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:23.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:23.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:23.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:23.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:23.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 28 >= 31 )) 2026-02-21T01:39:23.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:26.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:26.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:26.113 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 29 2026-02-21T01:39:26.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 29' 2026-02-21T01:39:26.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:26.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:26.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:26.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:26.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:26.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:26.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:26.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 29 >= 31 )) 2026-02-21T01:39:26.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:29.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:29.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:29.203 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 30 2026-02-21T01:39:29.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 30' 2026-02-21T01:39:29.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:29.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:29.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:29.204 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:29.204 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:29.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:29.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:29.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 30 >= 31 )) 2026-02-21T01:39:29.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 3 2026-02-21T01:39:32.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T01:39:32.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T01:39:32.288 INFO:tasks.workunit.client.0.trial106.stdout:#---------- 1.0 loop 31 2026-02-21T01:39:32.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 31' 2026-02-21T01:39:32.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T01:39:32.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T01:39:32.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T01:39:32.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T01:39:32.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T01:39:32.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T01:39:32.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T01:39:32.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 31 >= 31 )) 2026-02-21T01:39:32.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1712: wait_for_pg_clean: ceph report 2026-02-21T01:39:32.644 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_fingerprint": "dfee45d2-bb71-4467-9a13-86c2c2cc584b", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "version": "20.2.0-681-g36d388f2", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "commit": "36d388f27052dd940bdbaf543bb05bdba0acc155", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "timestamp": "2026-02-21T01:39:32.650777+0000", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "tag": "", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "health": { 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "status": "HEALTH_WARN", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "checks": { 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "OSD_DOWN": { 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:39:32.645 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "message": "1 osds down", 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "count": 1 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "message": "osd.0 (root=default,host=trial106) is down" 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "OSD_HOST_DOWN": { 2026-02-21T01:39:32.646 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "message": "1 host (1 osds) down", 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "count": 1 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "message": "host trial106 (root=default) (1 osds) is down" 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.647 INFO:tasks.workunit.client.0.trial106.stdout: "OSD_ROOT_DOWN": { 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "message": "1 root (1 osds) down", 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "count": 1 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "message": "root default (1 osds) is down" 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.648 INFO:tasks.workunit.client.0.trial106.stdout: "PG_AVAILABILITY": { 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "message": "Reduced data availability: 4 pgs stale", 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "count": 4 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.0 is stuck stale for 91s, current state stale+active+clean, last acting [0]" 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.1 is stuck stale for 91s, current state stale+active+clean, last acting [0]" 2026-02-21T01:39:32.649 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.2 is stuck stale for 91s, current state stale+active+clean, last acting [0]" 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.3 is stuck stale for 91s, current state stale+active+clean, last acting [0]" 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: "POOL_NO_REDUNDANCY": { 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:39:32.650 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "message": "1 pool(s) have no replicas configured", 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "count": 1 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pool 'rbd' has no replicas configured" 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "TOO_FEW_OSDS": { 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:39:32.651 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "count": 2 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [], 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "mutes": [] 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "monmap_first_committed": 1, 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "monmap_last_committed": 1, 2026-02-21T01:39:32.652 INFO:tasks.workunit.client.0.trial106.stdout: "monmap": { 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "fsid": "640ad1de-49d5-4d28-b121-3f053a140605", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:22:54.573410Z", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "created": "2026-02-21T01:22:54.573410Z", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "min_mon_release": 20, 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "min_mon_release_name": "tentacle", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "election_strategy": 1, 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "disallowed_leaders": "", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode": false, 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "tiebreaker_mon": "", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "removed_ranks": "", 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "features": { 2026-02-21T01:39:32.653 INFO:tasks.workunit.client.0.trial106.stdout: "persistent": [ 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "kraken", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "luminous", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "mimic", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap-prune", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "nautilus", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "octopus", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "pacific", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "elector-pinging", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "quincy", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "reef", 2026-02-21T01:39:32.654 INFO:tasks.workunit.client.0.trial106.stdout: "squid", 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "tentacle", 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "nvmeof_beacon_diff" 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "optional": [] 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "mons": [ 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "rank": 0, 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "name": "a", 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "public_addrs": { 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:39:32.655 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:7109", 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 0 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:7109/0", 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "public_addr": "127.0.0.1:7109/0", 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "priority": 0, 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 0, 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: "crush_location": "{}" 2026-02-21T01:39:32.656 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "quorum": [ 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: 0 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap": { 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 13, 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "fsid": "640ad1de-49d5-4d28-b121-3f053a140605", 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "created": "2026-02-21T01:22:54.650257+0000", 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:37:59.658048+0000", 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "last_up_change": "2026-02-21T01:37:59.658048+0000", 2026-02-21T01:39:32.657 INFO:tasks.workunit.client.0.trial106.stdout: "last_in_change": "2026-02-21T01:22:54.967844+0000", 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "flags_num": 5799936, 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "flags_set": [ 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "pglog_hardlimit", 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snapdirs", 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "recovery_deletes", 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "sortbitwise" 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "crush_version": 3, 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "full_ratio": 0.99000000953674316, 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "backfillfull_ratio": 0.99000000953674316, 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "nearfull_ratio": 0.85000002384185791, 2026-02-21T01:39:32.658 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_snapshot": "", 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "pool_max": 1, 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "max_osd": 1, 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "require_min_compat_client": "luminous", 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "min_compat_client": "jewel", 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "require_osd_release": "tentacle", 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "allow_crimson": false, 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "pools": [ 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "pool_name": "rbd", 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "create_time": "2026-02-21T01:22:58.794431+0000", 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "flags": 8193, 2026-02-21T01:39:32.659 INFO:tasks.workunit.client.0.trial106.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "type": 1, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "size": 1, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "min_size": 1, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "crush_rule": 0, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_count": 0, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_target": 0, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_barrier": 0, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "is_stretch_pool": false, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "object_hash": 2, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "pg_autoscale_mode": "off", 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num": 4, 2026-02-21T01:39:32.660 INFO:tasks.workunit.client.0.trial106.stdout: "pg_placement_num": 4, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "pg_placement_num_target": 4, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num_target": 4, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num_pending": 4, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "last_pg_merge_meta": { 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "source_pgid": "0.0", 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "ready_epoch": 0, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_started": 0, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_clean": 0, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "source_version": "0'0", 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "target_version": "0'0" 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "last_change": "11", 2026-02-21T01:39:32.661 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend": "0", 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend_prenautilus": "0", 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend_preluminous": "0", 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "auid": 0, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "snap_mode": "selfmanaged", 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "snap_seq": 2, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "snap_epoch": 11, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "pool_snaps": [], 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "removed_snaps": "[]", 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "quota_max_bytes": 0, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "quota_max_objects": 0, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "tiers": [], 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "tier_of": -1, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "read_tier": -1, 2026-02-21T01:39:32.662 INFO:tasks.workunit.client.0.trial106.stdout: "write_tier": -1, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "cache_mode": "none", 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "target_max_bytes": 0, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "target_max_objects": 0, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_dirty_ratio_micro": 400000, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_full_ratio_micro": 800000, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "cache_min_flush_age": 0, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "cache_min_evict_age": 0, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "erasure_code_profile": "", 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_params": { 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "type": "none" 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.663 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_period": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_count": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "use_gmt_hitset": true, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "min_read_recency_for_promote": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "min_write_recency_for_promote": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_grade_decay_rate": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_search_last_n": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "grade_table": [], 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "stripe_width": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "expected_num_objects": 0, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "fast_read": false, 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "nonprimary_shards": "{}", 2026-02-21T01:39:32.664 INFO:tasks.workunit.client.0.trial106.stdout: "options": {}, 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "application_metadata": { 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "rbd": {} 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "osds": [ 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "uuid": "3a40b190-6837-4acb-90dd-7b79d6cf1494", 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "up": 0, 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "in": 1, 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 1, 2026-02-21T01:39:32.665 INFO:tasks.workunit.client.0.trial106.stdout: "primary_affinity": 1, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "last_clean_begin": 0, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "last_clean_end": 0, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 5, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "up_thru": 6, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "down_at": 13, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "lost_at": 0, 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "public_addrs": { 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6800", 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.666 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6801", 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_addrs": { 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6802", 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.667 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6803", 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_back_addrs": { 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:39:32.668 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6806", 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6807", 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_front_addrs": { 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.669 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6804", 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6805", 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3580950639 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "public_addr": "127.0.0.1:6801/3580950639", 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_addr": "127.0.0.1:6803/3580950639", 2026-02-21T01:39:32.670 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_back_addr": "127.0.0.1:6807/3580950639", 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_front_addr": "127.0.0.1:6805/3580950639", 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "state": [ 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "exists" 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "osd_xinfo": [ 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "down_stamp": "2026-02-21T01:37:59.658048+0000", 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "laggy_probability": 0, 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "laggy_interval": 0, 2026-02-21T01:39:32.671 INFO:tasks.workunit.client.0.trial106.stdout: "features": 4544132024016699391, 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "old_weight": 0, 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "last_purged_snaps_scrub": "0.000000", 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "dead_epoch": 0 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap": [], 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap_items": [], 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap_primaries": [], 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "pg_temp": [], 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "primary_temp": [], 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "blocklist": {}, 2026-02-21T01:39:32.672 INFO:tasks.workunit.client.0.trial106.stdout: "range_blocklist": {}, 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "erasure_code_profiles": { 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "default": { 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "k": "2", 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "m": "2", 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "plugin": "isa", 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "technique": "reed_sol_van" 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "removed_snaps_queue": [], 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "new_removed_snaps": [], 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "new_purged_snaps": [], 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "crush_node_flags": {}, 2026-02-21T01:39:32.673 INFO:tasks.workunit.client.0.trial106.stdout: "device_class_flags": {}, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode": { 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode_enabled": false, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_bucket_count": 0, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "degraded_stretch_mode": 0, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "recovering_stretch_mode": 0, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode_bucket": 0 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "osd_metadata": [ 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "arch": "x86_64", 2026-02-21T01:39:32.674 INFO:tasks.workunit.client.0.trial106.stdout: "back_addr": "[v2:127.0.0.1:6802/3580950639,v1:127.0.0.1:6803/3580950639]", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "back_iface": "", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs": "1", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_dedicated_db": "0", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_dedicated_wal": "0", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_single_shared_device": "1", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_allocation_from_file": "1", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_allocator": "hybrid", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_access_mode": "file", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_block_size": "4096", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_devices": "nvme0n1", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_driver": "KernelDevice", 2026-02-21T01:39:32.675 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_optimal_io_size": "4096", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_rotational": "0", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_size": "107374182400", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_support_discard": "1", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_type": "ssd", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_min_alloc_size": "4096", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_onode_segmentation": "inactive", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_write_mode": "classic", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_release": "tentacle", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version": "ceph version 20.2.0-681-g36d388f2 (36d388f27052dd940bdbaf543bb05bdba0acc155) tentacle (stable - None)", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version_short": "20.2.0-681-g36d388f2", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version_when_created": "ceph version 20.2.0-681-g36d388f2 (36d388f27052dd940bdbaf543bb05bdba0acc155) tentacle (stable - None)", 2026-02-21T01:39:32.676 INFO:tasks.workunit.client.0.trial106.stdout: "cpu": "AMD EPYC 4464P 12-Core Processor", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "created_at": "2026-02-21T01:22:55.595320Z", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "default_device_class": "ssd", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "device_ids": "nvme0n1=Micron_7450_MTFDKBA960TFR_25174FCD6E72", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "devices": "nvme0n1", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "distro": "ubuntu", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "distro_description": "Ubuntu 22.04.5 LTS", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "distro_version": "22.04", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "front_addr": "[v2:127.0.0.1:6800/3580950639,v1:127.0.0.1:6801/3580950639]", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "front_iface": "", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/3580950639,v1:127.0.0.1:6807/3580950639]", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/3580950639,v1:127.0.0.1:6805/3580950639]", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "hostname": "trial106", 2026-02-21T01:39:32.677 INFO:tasks.workunit.client.0.trial106.stdout: "journal_rotational": "0", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "kernel_description": "#180-Ubuntu SMP Fri Jan 9 16:10:31 UTC 2026", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "kernel_version": "5.15.0-170-generic", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "mem_swap_kb": "8388604", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "mem_total_kb": "130961696", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "objectstore_numa_node": "0", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "objectstore_numa_nodes": "0", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "os": "Linux", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "osd_data": "td/ceph-helpers/0", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "osd_objectstore": "bluestore", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "osdspec_affinity": "", 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: "rotational": "0" 2026-02-21T01:39:32.678 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_clean_epochs": { 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "min_last_epoch_clean": 0, 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_clean": { 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "per_pool": [] 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "osd_epochs": [] 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_first_committed": 1, 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_last_committed": 13, 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "crushmap": { 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: "devices": [ 2026-02-21T01:39:32.679 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "name": "osd.0", 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "class": "ssd" 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "types": [ 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 0, 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "name": "osd" 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.680 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "name": "host" 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 2, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "name": "chassis" 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 3, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "name": "rack" 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 4, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: "name": "row" 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.681 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 5, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "name": "pdu" 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 6, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "name": "pod" 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 7, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "name": "room" 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 8, 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: "name": "datacenter" 2026-02-21T01:39:32.682 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 9, 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "name": "zone" 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 10, 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "name": "region" 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "name": "root" 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.683 INFO:tasks.workunit.client.0.trial106.stdout: "buckets": [ 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "id": -1, 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "name": "default", 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "root", 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "id": -3, 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.684 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "id": -2, 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "name": "default~ssd", 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "root", 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "id": -4, 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.685 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "id": -3, 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "name": "trial106", 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "host", 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.686 INFO:tasks.workunit.client.0.trial106.stdout: "id": -4, 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "name": "trial106~ssd", 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "host", 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: "rules": [ 2026-02-21T01:39:32.687 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "rule_id": 0, 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "rule_name": "replicated_rule", 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "type": 1, 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "steps": [ 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "op": "take", 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "item": -1, 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "item_name": "default" 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "op": "choose_firstn", 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "num": 0, 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "type": "osd" 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: "op": "emit" 2026-02-21T01:39:32.688 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "tunables": { 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "choose_local_tries": 0, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "choose_local_fallback_tries": 0, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "choose_total_tries": 50, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_descend_once": 1, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_vary_r": 1, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_stable": 1, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "msr_descents": 100, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "msr_collision_tries": 100, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "straw_calc_version": 1, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "allowed_bucket_algs": 54, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "profile": "jewel", 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "optimal_tunables": 1, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "legacy_tunables": 0, 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "minimum_required_version": "jewel", 2026-02-21T01:39:32.689 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables": 1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables2": 1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "has_v2_rules": 0, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables3": 1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "has_v3_rules": 0, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "has_v4_buckets": 1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables5": 1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "has_v5_rules": 0, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "has_msr_rules": 0 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "choose_args": {} 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "fsmap": { 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "btime": "2026-02-21T01:22:54:650052+0000", 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "default_fscid": -1, 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "compat": { 2026-02-21T01:39:32.690 INFO:tasks.workunit.client.0.trial106.stdout: "compat": {}, 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "ro_compat": {}, 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "incompat": { 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_1": "base v0.20", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_2": "client writeable ranges", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_3": "default file layouts on dirs", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_4": "dir inode in separate object", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_5": "mds uses versioned encoding", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_6": "dirfrag is stored in omap", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_8": "no anchor table", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_9": "file layout v2", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_10": "snaprealm v2", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_11": "minor log segments", 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_12": "quiesce subvolumes" 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "feature_flags": { 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "enable_multiple": true, 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: "ever_enabled_multiple": true 2026-02-21T01:39:32.691 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "standbys": [], 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "filesystems": [] 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "mdsmap_first_committed": 1, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "mdsmap_last_committed": 1, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "auth": { 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "first_committed": 1, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "last_committed": 3, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_secrets": 1 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg": 4, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_active": 4, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_unknown": 0, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_osd": 1, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "pool_sum": { 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "stat_sum": { 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes": 19, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects": 2, 2026-02-21T01:39:32.692 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_clones": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_copies": 2, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing_on_primary": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_degraded": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_misplaced": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_unfound": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_dirty": 2, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_whiteouts": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_read": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_read_kb": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_write": 2, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_write_kb": 2, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_scrub_errors": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_shallow_scrub_errors": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_deep_scrub_errors": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_recovered": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_recovered": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_keys_recovered": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_omap": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_hit_set_archive": 0, 2026-02-21T01:39:32.693 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_hit_set_archive": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_kb": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_kb": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_promote": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_high": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_low": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_some": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_full": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_pinned": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_legacy_snapsets": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_large_omap_objects": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_manifest": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_bytes": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_keys": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_repaired": 0 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "store_stats": { 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "total": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "available": 0, 2026-02-21T01:39:32.694 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 0 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "log_size": 3, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "ondisk_log_size": 3, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "up": 4, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "acting": 4, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "num_store_stats": 0 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "osd_sum": { 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 0, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "num_pgs": 4, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "num_osds": 1, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_osds": 1, 2026-02-21T01:39:32.695 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_omap_osds": 1, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "kb": 104857600, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used": 27068, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_data": 108, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_omap": 6, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_meta": 26937, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "kb_avail": 104830532, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "statfs": { 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "total": 107374182400, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "available": 107346464768, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 110592, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 23248, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 6909, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 27583747 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "hb_peers": [], 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "snap_trim_queue_len": 0, 2026-02-21T01:39:32.696 INFO:tasks.workunit.client.0.trial106.stdout: "num_snap_trimming": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "num_shards_repaired": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "op_queue_age_hist": { 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "histogram": [], 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "upper_bound": 1 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "perf_stat": { 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ms": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ms": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ns": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ns": 0 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "alerts": [], 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "network_ping_times": [] 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "osd_sum_by_class": { 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "ssd": { 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 0, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "num_pgs": 4, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "num_osds": 1, 2026-02-21T01:39:32.697 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_osds": 1, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_omap_osds": 1, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "kb": 104857600, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used": 27068, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_data": 108, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_omap": 6, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_meta": 26937, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "kb_avail": 104830532, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "statfs": { 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "total": 107374182400, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "available": 107346464768, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 110592, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 23248, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 6909, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 27583747 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "hb_peers": [], 2026-02-21T01:39:32.698 INFO:tasks.workunit.client.0.trial106.stdout: "snap_trim_queue_len": 0, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "num_snap_trimming": 0, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "num_shards_repaired": 0, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "op_queue_age_hist": { 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "histogram": [], 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "upper_bound": 1 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "perf_stat": { 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ms": 0, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ms": 0, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ns": 0, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ns": 0 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "alerts": [], 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "network_ping_times": [] 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "pool_stats": [ 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "poolid": 1, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg": 4, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "stat_sum": { 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes": 19, 2026-02-21T01:39:32.699 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects": 2, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_clones": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_copies": 2, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing_on_primary": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_degraded": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_misplaced": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_unfound": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_dirty": 2, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_whiteouts": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_read": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_read_kb": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_write": 2, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_write_kb": 2, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_scrub_errors": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_shallow_scrub_errors": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_deep_scrub_errors": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_recovered": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_recovered": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_keys_recovered": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_omap": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_hit_set_archive": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_hit_set_archive": 0, 2026-02-21T01:39:32.700 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_kb": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_kb": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_promote": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_high": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_low": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_some": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_full": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_pinned": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_legacy_snapsets": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_large_omap_objects": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_manifest": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_bytes": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_keys": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_repaired": 0 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "store_stats": { 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "total": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "available": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 4096, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 19, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:39:32.701 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 0, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 0 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "log_size": 3, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "ondisk_log_size": 3, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "up": 4, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "acting": 4, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "num_store_stats": 1 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "osd_stats": [ 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 21474836487 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_by_state": [ 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "state": "stale+active+clean", 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "num": 4 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "pool_pg_unavailable_map": [ 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.702 INFO:tasks.workunit.client.0.trial106.stdout: "poolid": "1", 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "pgs": [ 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "1.3", 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "1.2", 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "1.1", 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "1.0" 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_by_osd": [ 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "num_primary_pg": 4, 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "num_acting_pg": 4, 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "num_up_not_acting_pg": 0 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snaps": [ 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snaps": {} 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "servicemap": { 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 3, 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:38:59.260944+0000", 2026-02-21T01:39:32.703 INFO:tasks.workunit.client.0.trial106.stdout: "services": {} 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "mgrstat_first_committed": 253, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "mgrstat_last_committed": 499, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "logm_first_committed": 1, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "logm_last_committed": 549, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "paxos": { 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "first_committed": 503, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "last_committed": 1073, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "last_pn": 0, 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: "accepted_pn": 0 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stderr:report 4260740572 2026-02-21T01:39:32.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1713: wait_for_pg_clean: echo 'PG 1.0 is not clean after 31 iterations' 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stdout:PG 1.0 is not clean after 31 iterations 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_pg_clean: return 1 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: return 1 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: test_pg_scrub: teardown td/ceph-helpers 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:39:32.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:32.706 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:32.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:32.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:32.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:39:32.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:39:32.761 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:39:32.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:39:32.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:39:32.763 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:39:32.764 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:39:32.764 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:39:32.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:32.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:39:32.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:39:32.766 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:32.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:39:32.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:39:32.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:39:32.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:39:32.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:32.781 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:32.781 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:39:32.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:39:32.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:39:32.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_repair td/ceph-helpers 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: test_repair: local dir=td/ceph-helpers 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: test_repair: setup td/ceph-helpers 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:32.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:39:32.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:39:32.786 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:39:32.786 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:39:32.787 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:39:32.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:39:32.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:39:32.788 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:39:32.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:32.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:39:32.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:39:32.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:32.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:39:32.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:39:32.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:39:32.793 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:39:32.793 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:32.794 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:32.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:39:32.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:39:32.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:39:32.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:39:32.796 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:39:32.796 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:32.796 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:32.796 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:39:32.798 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:39:32.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:39:32.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:39:32.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:39:32.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:39:32.820 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:32.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:39:32.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:39:32.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:39:32.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:39:32.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:39:32.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:39:32.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:39:32.849 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:39:32.849 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:32.849 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:32.849 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:32.849 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:32.849 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:32.850 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:32.850 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:39:32.850 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:39:32.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:39:32.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:39:32.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:39:32.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:39:32.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:32.907 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:32.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:39:32.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1898: test_repair: run_mgr td/ceph-helpers x 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:39:32.963 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:39:33.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:39:33.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:33.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:33.078 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:33.078 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:33.079 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:33.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:33.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:39:33.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:39:33.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1899: test_repair: run_osd td/ceph-helpers 0 2026-02-21T01:39:33.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:39:33.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:39:33.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:39:33.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:39:33.100 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:39:33.101 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:39:33.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:39:33.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:39:33.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:39:33.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:39:33.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:39:33.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:39:33.103 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:39:33.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f0a5764a-6589-4754-af88-fb29fc95a626 2026-02-21T01:39:33.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f0a5764a-6589-4754-af88-fb29fc95a626' 2026-02-21T01:39:33.103 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 f0a5764a-6589-4754-af88-fb29fc95a626 2026-02-21T01:39:33.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:39:33.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVDJlphNMhBxAAhvMMSZ0BDk0aKjV3AYZu1w== 2026-02-21T01:39:33.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVDJlphNMhBxAAhvMMSZ0BDk0aKjV3AYZu1w=="}' 2026-02-21T01:39:33.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f0a5764a-6589-4754-af88-fb29fc95a626 -i td/ceph-helpers/0/new.json 2026-02-21T01:39:33.220 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:39:33.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:39:33.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVDJlphNMhBxAAhvMMSZ0BDk0aKjV3AYZu1w== --osd-uuid f0a5764a-6589-4754-af88-fb29fc95a626 2026-02-21T01:39:33.244 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:33.246+0000 7efd3ad46a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:33.245 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:33.250+0000 7efd3ad46a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:33.246 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:33.250+0000 7efd3ad46a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:33.247 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:33.250+0000 7efd3ad46a40 -1 bdev(0x55bfeb320c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:39:33.247 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:33.250+0000 7efd3ad46a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:39:34.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:39:34.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:39:34.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:39:34.034 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:39:34.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:39:34.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:39:34.156 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:39:34.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:39:34.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:39:34.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:39:34.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:39:34.180 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:34.182+0000 7f71c033da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:34.182 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:34.186+0000 7f71c033da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:34.183 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:34.186+0000 7f71c033da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:34.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:39:34.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:39:34.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:39:34.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:39:34.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:39:34.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:34.268 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:39:34.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:39:34.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:34.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:39:34.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:34.719 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:34.722+0000 7f71c033da40 -1 Falling back to public interface 2026-02-21T01:39:35.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:35.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:35.378 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:39:35.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:39:35.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:35.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:39:35.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:35.635 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:35.638+0000 7f71c033da40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:39:36.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:36.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:36.592 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:39:36.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:39:36.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:36.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:39:36.813 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/102240903,v1:127.0.0.1:6801/102240903] [v2:127.0.0.1:6802/102240903,v1:127.0.0.1:6803/102240903] exists,up f0a5764a-6589-4754-af88-fb29fc95a626 2026-02-21T01:39:36.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:39:36.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:39:36.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:39:36.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1900: test_repair: create_rbd_pool 2026-02-21T01:39:36.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:39:37.013 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:39:37.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:39:37.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:39:37.257 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:39:37.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:39:38.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:39:38.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1901: test_repair: wait_for_clean 2026-02-21T01:39:38.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:39:38.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:39:38.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:39:38.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:39:38.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:39:38.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:39:38.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:39:38.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:39:38.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:39:38.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:39:38.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:39:38.865 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:39:38.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:39:38.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:39:38.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:39:38.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:39:38.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:39:38.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:39:38.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:39:38.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:39:38.946 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:39:38.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:39:38.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:39:38.948 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:39:38.948 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:39:39.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:39:39.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:39:40.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:39:40.164 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:39:40.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:39:40.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:39:40.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:39:40.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:39:40.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:39:40.646 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:39:40.847 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:39:40.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:39:40.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:39:40.848 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1902: test_repair: repair 1.0 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:39:41.123 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:39:41.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-21T01:39:37.210966+0000 2026-02-21T01:39:41.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-21T01:39:41.526 INFO:tasks.workunit.client.0.trial106.stderr:instructing pg 1.0 on osd.0 to repair 2026-02-21T01:39:41.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-21T01:39:37.210966+0000 2026-02-21T01:39:41.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-21T01:39:37.210966+0000 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:39:41.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:39:41.539 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:39:41.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:39:37.210966+0000 '>' 2026-02-21T01:39:37.210966+0000 2026-02-21T01:39:41.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:39:42.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:39:42.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:39:42.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:39:42.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:39:42.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:39:42.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:39:42.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:39:42.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:39:41.531976+0000 '>' 2026-02-21T01:39:37.210966+0000 2026-02-21T01:39:42.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-21T01:39:42.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1903: test_repair: kill_daemons td/ceph-helpers KILL osd 2026-02-21T01:39:42.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:42.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:42.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:42.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:42.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1904: test_repair: TIMEOUT=2 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1904: test_repair: repair 1.0 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:39:43.079 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:39:43.286 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-21T01:39:41.531976+0000 2026-02-21T01:39:43.287 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-21T01:39:43.487 INFO:tasks.workunit.client.0.trial106.stderr:instructing pg 1.0 on osd.0 to repair 2026-02-21T01:39:43.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-21T01:39:41.531976+0000 2026-02-21T01:39:43.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-21T01:39:41.531976+0000 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 2 )) 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:39:43.499 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:39:43.500 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:39:43.500 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:39:43.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:39:41.531976+0000 '>' 2026-02-21T01:39:41.531976+0000 2026-02-21T01:39:43.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:39:44.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:39:44.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 2 )) 2026-02-21T01:39:44.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:39:44.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:39:44.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:39:44.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:39:44.715 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:39:44.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:39:41.531976+0000 '>' 2026-02-21T01:39:41.531976+0000 2026-02-21T01:39:44.933 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:39:45.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:39:45.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 2 )) 2026-02-21T01:39:45.935 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: return 1 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1905: test_repair: teardown td/ceph-helpers 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:45.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:45.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:39:46.043 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:39:46.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:39:46.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:39:46.044 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:39:46.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:39:46.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:39:46.046 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:39:46.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:46.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:39:46.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:39:46.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:46.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:39:46.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:39:46.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:39:46.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:39:46.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:46.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:46.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_run_in_background td/ceph-helpers 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2198: test_run_in_background: local pids 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: test_run_in_background: run_in_background pids sleep 1 2026-02-21T01:39:46.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 117468"' 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 117468' 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: test_run_in_background: run_in_background pids sleep 1 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 117469"' 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 117469' 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sleep 1 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/117471: /' 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sleep 1 2026-02-21T01:39:46.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/117474: /' 2026-02-21T01:39:46.060 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_run_in_background: echo 117468 117469 2026-02-21T01:39:46.060 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_run_in_background: wc -w 2026-02-21T01:39:46.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: test_run_in_background: test 2 = 2 2026-02-21T01:39:46.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: test_run_in_background: wait 117468 117469 2026-02-21T01:39:47.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-21T01:39:47.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_run_mon td/ceph-helpers 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: test_run_mon: local dir=td/ceph-helpers 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:488: test_run_mon: setup td/ceph-helpers 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:39:47.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:39:47.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:39:47.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:47.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:47.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:47.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:47.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:39:47.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:39:47.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:39:47.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:39:47.065 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:39:47.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:39:47.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:39:47.067 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:39:47.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:47.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:39:47.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:39:47.069 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:39:47.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:39:47.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:39:47.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:39:47.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:39:47.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.072 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:39:47.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:39:47.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:39:47.073 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:39:47.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:39:47.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.075 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.075 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:39:47.076 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:490: test_run_mon: run_mon td/ceph-helpers a 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:39:47.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:39:47.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:39:47.098 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:47.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:47.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:47.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.099 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:47.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:39:47.125 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:39:47.127 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:47.128 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:47.129 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.129 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.129 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:47.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:39:47.129 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:39:47.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:39:47.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:39:47.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:39:47.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:39:47.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:39:47.185 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:39:47.185 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:47.185 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:47.186 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:47.186 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.186 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.186 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:47.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:39:47.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:39:47.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:491: test_run_mon: ceph mon dump 2026-02-21T01:39:47.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:491: test_run_mon: grep mon.a 2026-02-21T01:39:47.347 INFO:tasks.workunit.client.0.trial106.stderr:dumped monmap epoch 1 2026-02-21T01:39:47.354 INFO:tasks.workunit.client.0.trial106.stdout:0: v2:127.0.0.1:7109/0 mon.a 2026-02-21T01:39:47.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:492: test_run_mon: kill_daemons td/ceph-helpers 2026-02-21T01:39:47.355 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:47.355 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:47.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:47.355 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:47.356 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:39:47.463 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:494: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:39:47.464 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2026-02-21T01:39:47.477 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:47.478+0000 7f0735d9aec0 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2026-02-21T01:39:47.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:39:47.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:47.479 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:47.479 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:47.480 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.480 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.480 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:47.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-21T01:39:47.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:39:47.511 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:47.512 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:47.512 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:47.512 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.512 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.512 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:47.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:39:47.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:39:47.568 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.569 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.570 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:47.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:39:47.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: run_osd td/ceph-helpers 0 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:47.624 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:39:47.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:39:47.627 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:39:47.628 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8fae08ed-976e-4341-938d-2b01030b6ed0 2026-02-21T01:39:47.628 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8fae08ed-976e-4341-938d-2b01030b6ed0' 2026-02-21T01:39:47.628 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 8fae08ed-976e-4341-938d-2b01030b6ed0 2026-02-21T01:39:47.628 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:39:47.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjDJlpaTKVJhAA1MH/TsvlxwKIpU7ziS/eRA== 2026-02-21T01:39:47.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjDJlpaTKVJhAA1MH/TsvlxwKIpU7ziS/eRA=="}' 2026-02-21T01:39:47.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8fae08ed-976e-4341-938d-2b01030b6ed0 -i td/ceph-helpers/0/new.json 2026-02-21T01:39:47.762 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:39:47.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:39:47.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjDJlpaTKVJhAA1MH/TsvlxwKIpU7ziS/eRA== --osd-uuid 8fae08ed-976e-4341-938d-2b01030b6ed0 2026-02-21T01:39:47.797 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:47.798+0000 7fd217883a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:47.798 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:47.802+0000 7fd217883a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:47.799 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:47.802+0000 7fd217883a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:47.800 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:47.802+0000 7fd217883a40 -1 bdev(0x55d733638c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:39:47.800 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:47.802+0000 7fd217883a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:39:48.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:39:48.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:39:48.399 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:39:48.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:39:48.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:39:48.522 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:39:48.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:39:48.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:39:48.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:39:48.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:39:48.523 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:39:48.546 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:48.550+0000 7ff230deca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:48.549 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:48.550+0000 7ff230deca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:48.550 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:48.554+0000 7ff230deca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:48.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:48.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:39:48.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:48.895 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:48.898+0000 7ff230deca40 -1 Falling back to public interface 2026-02-21T01:39:49.461 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:49.462+0000 7ff230deca40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:39:49.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:49.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:49.747 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:39:49.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:39:49.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:49.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:39:49.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:50.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:50.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:50.861 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:39:50.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:39:50.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:50.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:39:50.968 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3811998913,v1:127.0.0.1:6801/3811998913] [v2:127.0.0.1:6802/3811998913,v1:127.0.0.1:6803/3811998913] exists,up 8fae08ed-976e-4341-938d-2b01030b6ed0 2026-02-21T01:39:50.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:39:50.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:39:50.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:39:50.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:496: test_run_mon: run_osd td/ceph-helpers 1 2026-02-21T01:39:50.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:50.970 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:50.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:50.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:50.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:39:50.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:39:50.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:39:50.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:39:50.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9f744b78-8516-4ea7-930a-8b7aa0b058b3 2026-02-21T01:39:50.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 9f744b78-8516-4ea7-930a-8b7aa0b058b3' 2026-02-21T01:39:50.973 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 9f744b78-8516-4ea7-930a-8b7aa0b058b3 2026-02-21T01:39:50.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:39:50.985 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDmDJlpb64gOxAA4KTk/YxYapwAHkTGQPnitw== 2026-02-21T01:39:50.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDmDJlpb64gOxAA4KTk/YxYapwAHkTGQPnitw=="}' 2026-02-21T01:39:50.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9f744b78-8516-4ea7-930a-8b7aa0b058b3 -i td/ceph-helpers/1/new.json 2026-02-21T01:39:51.100 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:39:51.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:39:51.109 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmDJlpb64gOxAA4KTk/YxYapwAHkTGQPnitw== --osd-uuid 9f744b78-8516-4ea7-930a-8b7aa0b058b3 2026-02-21T01:39:51.125 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.126+0000 7f4849240a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:51.127 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.130+0000 7f4849240a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:51.128 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.130+0000 7f4849240a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:51.128 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.130+0000 7f4849240a40 -1 bdev(0x560d25ec4c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:39:51.129 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.130+0000 7f4849240a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:39:51.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:39:51.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:39:51.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:39:51.822 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:39:51.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:39:51.943 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:39:51.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:39:51.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:39:51.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:39:51.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:39:51.943 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:39:51.967 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.970+0000 7f287b893a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:51.969 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.970+0000 7f287b893a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:51.970 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:51.974+0000 7f287b893a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:52.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:39:52.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:52.327 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:52.330+0000 7f287b893a40 -1 Falling back to public interface 2026-02-21T01:39:52.932 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:52.934+0000 7f287b893a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:39:53.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:53.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:53.168 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:39:53.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:39:53.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:53.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:39:53.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:54.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:54.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:54.280 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:39:54.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:39:54.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:54.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:39:54.388 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3774721871,v1:127.0.0.1:6809/3774721871] [v2:127.0.0.1:6810/3774721871,v1:127.0.0.1:6811/3774721871] exists,up 9f744b78-8516-4ea7-930a-8b7aa0b058b3 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: run_osd td/ceph-helpers 2 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/2 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:39:54.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:54.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:39:54.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/2 2026-02-21T01:39:54.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:39:54.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b8932d4b-85ef-4b6b-8760-853c2f6383e0 2026-02-21T01:39:54.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b8932d4b-85ef-4b6b-8760-853c2f6383e0' 2026-02-21T01:39:54.393 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 b8932d4b-85ef-4b6b-8760-853c2f6383e0 2026-02-21T01:39:54.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:39:54.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqDJlpL+GCGBAA0SzggrJr6hsmrZvNcjAg6g== 2026-02-21T01:39:54.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqDJlpL+GCGBAA0SzggrJr6hsmrZvNcjAg6g=="}' 2026-02-21T01:39:54.405 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b8932d4b-85ef-4b6b-8760-853c2f6383e0 -i td/ceph-helpers/2/new.json 2026-02-21T01:39:54.522 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:39:54.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/2/new.json 2026-02-21T01:39:54.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqDJlpL+GCGBAA0SzggrJr6hsmrZvNcjAg6g== --osd-uuid b8932d4b-85ef-4b6b-8760-853c2f6383e0 2026-02-21T01:39:54.547 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:54.549+0000 7f6c2ab34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:54.548 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:54.549+0000 7f6c2ab34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:54.549 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:54.549+0000 7f6c2ab34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:54.550 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:54.553+0000 7f6c2ab34a40 -1 bdev(0x5603b8bf4c00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:39:54.550 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:54.553+0000 7f6c2ab34a40 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2026-02-21T01:39:55.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/2/keyring 2026-02-21T01:39:55.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:39:55.202 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:39:55.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:39:55.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:39:55.324 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:39:55.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:39:55.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:39:55.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:39:55.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:39:55.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:39:55.347 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:55.349+0000 7f1249a66a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:55.349 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:55.349+0000 7f1249a66a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:55.350 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:55.353+0000 7f1249a66a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:39:55.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:39:55.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:39:55.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:39:55.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:39:55.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:39:55.438 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:55.438 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:39:55.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:39:55.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:55.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:39:55.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:55.751 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:55.753+0000 7f1249a66a40 -1 Falling back to public interface 2026-02-21T01:39:56.335 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:56.337+0000 7f1249a66a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:39:56.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:56.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:56.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:39:56.547 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:39:56.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:56.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:39:56.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:39:57.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:39:57.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:39:57.657 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:39:57.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:39:57.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:39:57.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:39:57.766 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/3026228291,v1:127.0.0.1:6817/3026228291] [v2:127.0.0.1:6818/3026228291,v1:127.0.0.1:6819/3026228291] exists,up b8932d4b-85ef-4b6b-8760-853c2f6383e0 2026-02-21T01:39:57.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:39:57.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:39:57.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:39:57.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:498: test_run_mon: create_rbd_pool 2026-02-21T01:39:57.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:39:57.870 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:39:57.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:39:57.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:39:58.011 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:39:58.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:39:59.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:39:59.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: ceph osd dump 2026-02-21T01:39:59.300 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: grep 'pool 1 '\''rbd'\''' 2026-02-21T01:39:59.412 INFO:tasks.workunit.client.0.trial106.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 20 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: get_asok_path mon.a 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:39:59.413 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:39:59.414 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: CEPH_ARGS= 2026-02-21T01:39:59.414 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get osd_pool_default_size 2026-02-21T01:39:59.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2026-02-21T01:39:59.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:502: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2026-02-21T01:39:59.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: CEPH_ARGS= 2026-02-21T01:39:59.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: ceph status 2026-02-21T01:39:59.539 INFO:tasks.workunit.client.0.trial106.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2026-02-21T01:39:59.539 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.541+0000 7ffac66e2640 -1 failed for service _ceph-mon._tcp 2026-02-21T01:39:59.539 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.541+0000 7ffac66e2640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2026-02-21T01:39:59.539 INFO:tasks.workunit.client.0.trial106.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2026-02-21T01:39:59.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: CEPH_ARGS= 2026-02-21T01:39:59.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2026-02-21T01:39:59.599 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-21T01:39:59.600 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 AuthRegistry(0x7f2c44052cd0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-21T01:39:59.600 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-21T01:39:59.600 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 AuthRegistry(0x7f2c490a0f50) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-21T01:39:59.601 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-21T01:39:59.601 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 AuthRegistry(0x7f2c4404ff10) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-21T01:39:59.601 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-21T01:39:59.601 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.601+0000 7f2c490a2640 -1 AuthRegistry(0x7f2c4412fce0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-21T01:39:59.601 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.605+0000 7f2c490a2640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2026-02-21T01:39:59.601 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:39:59.605+0000 7f2c490a2640 -1 AuthRegistry(0x7f2c4404ff10) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2026-02-21T01:39:59.649 INFO:tasks.workunit.client.0.trial106.stdout: cluster: 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: id: 640ad1de-49d5-4d28-b121-3f053a140605 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: health: HEALTH_OK 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: services: 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: mon: 1 daemons, quorum a (age 12s) [leader: a] 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: mgr: no daemons active 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: osd: 3 osds: 3 up (since 2s), 3 in (since 5s) 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: data: 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: pools: 0 pools, 0 pgs 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: objects: 0 objects, 0 B 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: usage: 0 B used, 0 B / 0 B avail 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: pgs: 2026-02-21T01:39:59.650 INFO:tasks.workunit.client.0.trial106.stdout: 2026-02-21T01:39:59.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: kill_daemons td/ceph-helpers 2026-02-21T01:39:59.661 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:39:59.661 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:39:59.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:39:59.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:39:59.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:04.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:04.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:509: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:40:04.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:40:04.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:40:04.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:40:04.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:40:04.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:40:04.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:40:04.990 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:04.993+0000 7faa589c2ec0 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2026-02-21T01:40:04.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:40:04.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:04.992 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:04.992 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:04.992 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:04.993 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:04.993 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:04.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:40:05.024 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:05.026 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:05.027 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:05.027 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.027 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.027 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:05.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:05.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:40:05.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:40:05.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:05.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:05.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:40:05.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:40:05.083 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:05.083 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:05.083 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:05.083 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:05.083 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.083 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.084 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:05.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:05.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:40:05.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: get_asok_path mon.a 2026-02-21T01:40:05.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:05.139 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:05.140 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:05.140 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.140 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.140 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:05.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: CEPH_ARGS= 2026-02-21T01:40:05.140 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get osd_pool_default_size 2026-02-21T01:40:05.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2026-02-21T01:40:05.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:512: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2026-02-21T01:40:05.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: kill_daemons td/ceph-helpers 2026-02-21T01:40:05.191 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:05.192 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:05.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:05.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:05.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:05.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:05.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:515: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2026-02-21T01:40:05.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:515: test_run_mon: run_mon td/ceph-helpers a 2026-02-21T01:40:05.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:40:05.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:40:05.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:40:05.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:40:05.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:40:05.501 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:40:05.514 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:05.517+0000 7f6067557ec0 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2026-02-21T01:40:05.516 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:40:05.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:05.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:05.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:05.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.517 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.517 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:05.518 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:40:05.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:40:05.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:40:05.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:05.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:05.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:40:05.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:05.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:05.548 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:40:05.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:40:05.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.602 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.603 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:05.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:05.603 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: get_asok_path mon.a 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:05.658 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: CEPH_ARGS= 2026-02-21T01:40:05.659 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get osd_pool_default_size 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:519: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: kill_daemons td/ceph-helpers 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:05.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:06.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: teardown td/ceph-helpers 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:06.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:06.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:06.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:06.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:06.029 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:06.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:06.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:06.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:06.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:06.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:06.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:06.033 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:06.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:06.035 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:06.035 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:06.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:06.042 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.042 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:06.043 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_run_osd td/ceph-helpers 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: test_run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:738: test_run_osd: setup td/ceph-helpers 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:06.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:06.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:06.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:06.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:06.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:06.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:06.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:06.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:06.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:06.048 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:06.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:06.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:06.049 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:06.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:06.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:06.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:06.050 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:06.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:06.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:06.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:06.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:06.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:06.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:06.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:06.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:06.056 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:06.057 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.057 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:06.058 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:06.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:740: test_run_osd: run_mon td/ceph-helpers a 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:40:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T01:40:06.081 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:40:06.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:06.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:06.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:06.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.082 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:06.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T01:40:06.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:40:06.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:40:06.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:06.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:06.115 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:40:06.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:40:06.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:06.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:06.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:40:06.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:40:06.172 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:06.173 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:06.173 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:06.173 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:06.173 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.173 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.173 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:06.174 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:06.174 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:40:06.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:741: test_run_osd: run_mgr td/ceph-helpers x 2026-02-21T01:40:06.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:40:06.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:40:06.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:40:06.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:40:06.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:40:06.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:40:06.344 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:40:06.344 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:06.344 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:06.345 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:06.345 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.345 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.345 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:06.345 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:40:06.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:40:06.368 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:743: test_run_osd: run_osd td/ceph-helpers 0 2026-02-21T01:40:06.368 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:06.368 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:40:06.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:06.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:06.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:40:06.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:06.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a763c0f-0669-41e8-9a68-97cf8a315feb 2026-02-21T01:40:06.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7a763c0f-0669-41e8-9a68-97cf8a315feb' 2026-02-21T01:40:06.372 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 7a763c0f-0669-41e8-9a68-97cf8a315feb 2026-02-21T01:40:06.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:06.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD2DJlpu1kJFxAAMs699qRKzseTSqNWDBPz/w== 2026-02-21T01:40:06.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD2DJlpu1kJFxAAMs699qRKzseTSqNWDBPz/w=="}' 2026-02-21T01:40:06.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a763c0f-0669-41e8-9a68-97cf8a315feb -i td/ceph-helpers/0/new.json 2026-02-21T01:40:06.490 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:06.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:40:06.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2DJlpu1kJFxAAMs699qRKzseTSqNWDBPz/w== --osd-uuid 7a763c0f-0669-41e8-9a68-97cf8a315feb 2026-02-21T01:40:06.514 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:06.517+0000 7fbede171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:06.515 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:06.517+0000 7fbede171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:06.516 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:06.517+0000 7fbede171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:06.516 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:06.517+0000 7fbede171a40 -1 bdev(0x5607ea3f4c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:06.517 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:06.517+0000 7fbede171a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:40:07.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:40:07.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:07.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:40:07.316 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:40:07.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:07.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:40:07.441 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:40:07.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:07.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:07.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:07.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:07.464 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:07.465+0000 7fab441c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:07.467 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:07.469+0000 7fab441c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:07.468 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:07.469+0000 7fab441c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:07.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:40:07.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:07.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:40:07.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:07.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:07.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:07.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:07.554 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:07.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:07.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:07.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:07.836 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:07.837+0000 7fab441c4a40 -1 Falling back to public interface 2026-02-21T01:40:08.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:08.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:08.665 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:08.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:08.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:08.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:08.803 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:08.805+0000 7fab441c4a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:40:08.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:09.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:09.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:09.891 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:09.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:40:09.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:09.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:10.104 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2164659413,v1:127.0.0.1:6801/2164659413] [v2:127.0.0.1:6802/2164659413,v1:127.0.0.1:6803/2164659413] exists,up 7a763c0f-0669-41e8-9a68-97cf8a315feb 2026-02-21T01:40:10.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:10.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:10.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:10.106 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: get_asok_path osd.0 2026-02-21T01:40:10.106 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T01:40:10.107 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T01:40:10.107 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:10.107 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:10.107 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:10.107 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.0.asok 2026-02-21T01:40:10.107 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: CEPH_ARGS= 2026-02-21T01:40:10.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T01:40:10.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-21T01:40:10.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:746: test_run_osd: echo '{"osd_max_backfills":"1"}' 2026-02-21T01:40:10.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:746: test_run_osd: grep --quiet osd_max_backfills 2026-02-21T01:40:10.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:748: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2026-02-21T01:40:10.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:10.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:10.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:10.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:10.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-max-backfills 20' 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:40:10.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:10.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=87eb34b4-2f64-401f-ac2a-9d258f3e73da 2026-02-21T01:40:10.173 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 87eb34b4-2f64-401f-ac2a-9d258f3e73da 2026-02-21T01:40:10.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 87eb34b4-2f64-401f-ac2a-9d258f3e73da' 2026-02-21T01:40:10.173 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:10.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6DJlpmkxHCxAAipd3CCZ4yIMs3URC77faWQ== 2026-02-21T01:40:10.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6DJlpmkxHCxAAipd3CCZ4yIMs3URC77faWQ=="}' 2026-02-21T01:40:10.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 87eb34b4-2f64-401f-ac2a-9d258f3e73da -i td/ceph-helpers/1/new.json 2026-02-21T01:40:10.397 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:10.407 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:40:10.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQD6DJlpmkxHCxAAipd3CCZ4yIMs3URC77faWQ== --osd-uuid 87eb34b4-2f64-401f-ac2a-9d258f3e73da 2026-02-21T01:40:10.424 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:10.425+0000 7fa1f4d5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:10.426 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:10.425+0000 7fa1f4d5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:10.427 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:10.429+0000 7fa1f4d5ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:10.427 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:10.429+0000 7fa1f4d5ba40 -1 bdev(0x5599c7c62c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:10.427 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:10.429+0000 7fa1f4d5ba40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:40:11.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:40:11.121 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:11.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:40:11.122 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:40:11.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:11.408 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:40:11.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:40:11.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-21T01:40:11.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:11.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:11.409 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:11.425 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:11.425+0000 7f9066dcda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:11.426 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:11.429+0000 7f9066dcda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:11.427 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:11.429+0000 7f9066dcda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:40:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:40:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:11.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:11.624 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:11.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:11.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:11.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:11.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:11.863 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:11.865+0000 7f9066dcda40 -1 Falling back to public interface 2026-02-21T01:40:12.468 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:12.469+0000 7f9066dcda40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:40:12.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:12.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:12.830 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:12.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:12.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:12.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:13.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:14.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:14.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:14.051 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:14.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:40:14.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:14.053 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:14.262 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/764349325,v1:127.0.0.1:6811/764349325] [v2:127.0.0.1:6812/764349325,v1:127.0.0.1:6813/764349325] exists,up 87eb34b4-2f64-401f-ac2a-9d258f3e73da 2026-02-21T01:40:14.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:14.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:14.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: get_asok_path osd.1 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:14.263 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.1.asok 2026-02-21T01:40:14.264 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: CEPH_ARGS= 2026-02-21T01:40:14.264 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.1.asok config get osd_max_backfills 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:751: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:753: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:753: test_run_osd: run_osd td/ceph-helpers 2 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/2 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:14.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:14.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:14.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/2 2026-02-21T01:40:14.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:14.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=91886891-f743-49cf-bc10-52ab81d0219c 2026-02-21T01:40:14.328 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 91886891-f743-49cf-bc10-52ab81d0219c 2026-02-21T01:40:14.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 91886891-f743-49cf-bc10-52ab81d0219c' 2026-02-21T01:40:14.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:14.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+DJlpVs+dFBAAUgolpXEJOySX+lZRb0I6NQ== 2026-02-21T01:40:14.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+DJlpVs+dFBAAUgolpXEJOySX+lZRb0I6NQ=="}' 2026-02-21T01:40:14.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 91886891-f743-49cf-bc10-52ab81d0219c -i td/ceph-helpers/2/new.json 2026-02-21T01:40:14.559 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:14.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/2/new.json 2026-02-21T01:40:14.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+DJlpVs+dFBAAUgolpXEJOySX+lZRb0I6NQ== --osd-uuid 91886891-f743-49cf-bc10-52ab81d0219c 2026-02-21T01:40:14.586 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:14.589+0000 7fda48e04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:14.587 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:14.589+0000 7fda48e04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:14.589 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:14.589+0000 7fda48e04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:14.589 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:14.589+0000 7fda48e04a40 -1 bdev(0x556ee624cc00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:14.589 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:14.589+0000 7fda48e04a40 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2026-02-21T01:40:15.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/2/keyring 2026-02-21T01:40:15.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:15.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:40:15.258 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:40:15.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:15.543 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:40:15.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:40:15.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:15.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:15.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:15.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:15.560 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:15.565+0000 7f2548132a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:15.562 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:15.569+0000 7f2548132a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:15.563 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:15.569+0000 7f2548132a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:15.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:40:15.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:15.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:40:15.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:15.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:15.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:15.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:15.750 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:15.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:15.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:40:15.920 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:15.924+0000 7f2548132a40 -1 Falling back to public interface 2026-02-21T01:40:15.967 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:16.572 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:16.576+0000 7f2548132a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:40:16.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:16.968 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:16.968 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:16.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:16.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:16.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:40:17.183 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/173439779,v1:127.0.0.1:6819/173439779] [v2:127.0.0.1:6820/173439779,v1:127.0.0.1:6821/173439779] exists,up 91886891-f743-49cf-bc10-52ab81d0219c 2026-02-21T01:40:17.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:17.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:17.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:17.185 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: get_asok_path osd.2 2026-02-21T01:40:17.185 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2026-02-21T01:40:17.185 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2026-02-21T01:40:17.186 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.186 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.186 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.186 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-osd.2.asok 2026-02-21T01:40:17.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: CEPH_ARGS= 2026-02-21T01:40:17.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.61441/ceph-osd.2.asok config get osd_max_backfills 2026-02-21T01:40:17.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2026-02-21T01:40:17.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:756: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:758: test_run_osd: teardown td/ceph-helpers 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:17.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:17.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:17.555 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:17.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:17.557 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:17.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:17.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:17.559 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:17.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:17.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:17.561 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:17.561 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:17.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:17.563 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:17.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:17.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:17.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:17.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:17.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:17.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_set_config td/ceph-helpers 2026-02-21T01:40:17.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: test_set_config: local dir=td/ceph-helpers 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1165: test_set_config: setup td/ceph-helpers 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:17.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:17.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:17.575 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:17.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:17.576 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:17.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:17.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:17.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:17.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:17.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:17.580 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:17.580 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:17.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:17.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:17.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:17.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:17.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.584 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:17.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:17.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:17.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:17.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:17.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.587 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:17.588 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:40:17.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:40:17.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:40:17.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:40:17.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:17.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:17.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:17.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.614 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:17.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:40:17.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.644 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.645 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.645 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:17.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:40:17.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:40:17.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.703 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.704 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:17.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:40:17.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: test_set_config: get_config mon a ms_crc_header 2026-02-21T01:40:17.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:17.760 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:17.760 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=ms_crc_header 2026-02-21T01:40:17.760 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .ms_crc_header 2026-02-21T01:40:17.760 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:17.760 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.760 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.761 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.761 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.761 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.762 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:17.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get ms_crc_header 2026-02-21T01:40:17.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: test_set_config: test true = true 2026-02-21T01:40:17.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: test_set_config: set_config mon a ms_crc_header false 2026-02-21T01:40:17.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=mon 2026-02-21T01:40:17.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=a 2026-02-21T01:40:17.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=ms_crc_header 2026-02-21T01:40:17.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=false 2026-02-21T01:40:17.819 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-21T01:40:17.820 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path mon.a 2026-02-21T01:40:17.820 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.820 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.820 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.820 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.821 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.821 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config set ms_crc_header false 2026-02-21T01:40:17.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: test_set_config: get_config mon a ms_crc_header 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=ms_crc_header 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .ms_crc_header 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:17.876 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:17.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get ms_crc_header 2026-02-21T01:40:17.930 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: test_set_config: test false = false 2026-02-21T01:40:17.930 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: test_set_config: set_config mon a ms_crc_header true 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=mon 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=a 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=ms_crc_header 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path mon.a 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.931 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.932 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.932 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config set ms_crc_header true 2026-02-21T01:40:17.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: test_set_config: get_config mon a ms_crc_header 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=ms_crc_header 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .ms_crc_header 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:17.986 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:17.987 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:17.987 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:17.987 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:17.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:17.987 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get ms_crc_header 2026-02-21T01:40:18.040 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: test_set_config: test true = true 2026-02-21T01:40:18.040 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: teardown td/ceph-helpers 2026-02-21T01:40:18.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:18.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:18.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:18.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:18.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:18.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:18.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:18.145 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:18.146 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:18.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:18.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:18.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:18.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:18.149 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:18.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:18.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:18.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:18.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:18.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:18.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:18.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_setup td/ceph-helpers 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: test_setup: local dir=td/ceph-helpers 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: test_setup: setup td/ceph-helpers 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:18.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:18.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:18.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:18.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:18.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:18.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:18.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:18.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:18.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:18.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:18.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:18.163 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:18.163 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:18.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:18.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:18.165 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:18.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:18.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:18.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:18.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:18.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:18.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:18.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:18.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:18.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.172 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:18.174 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:137: test_setup: test -d td/ceph-helpers 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: test_setup: setup td/ceph-helpers 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:18.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:18.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:18.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:18.179 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:18.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:18.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:18.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:18.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:18.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:18.182 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:18.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:18.184 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:18.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:18.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.186 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:18.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:18.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:18.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:18.188 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:18.188 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:18.190 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: test_setup: test -d td/ceph-helpers 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:140: test_setup: teardown td/ceph-helpers 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:18.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:18.193 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:18.193 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:18.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:18.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:18.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:18.194 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:18.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:18.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:18.196 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:18.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:18.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:18.197 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:18.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:18.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:18.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:18.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:18.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:18.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:18.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_teardown td/ceph-helpers 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:211: test_teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:212: test_teardown: setup td/ceph-helpers 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:18.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:18.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:18.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:18.207 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:18.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:18.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:18.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:18.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:18.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:18.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:18.211 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:18.211 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:18.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:18.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:18.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:18.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:18.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:18.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:18.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:18.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:18.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:18.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:213: test_teardown: teardown td/ceph-helpers 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:18.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:18.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:18.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:18.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:18.223 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:18.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:18.224 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:18.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:18.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:18.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:18.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:18.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:18.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:18.229 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:18.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:18.230 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:18.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:18.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:18.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:18.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:18.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:18.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:18.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:214: test_teardown: test -d td/ceph-helpers 2026-02-21T01:40:18.235 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_background td/ceph-helpers 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: test_wait_background: local pids= 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 130948"' 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 130948' 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-21T01:40:18.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 130949"' 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 130949' 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: test_wait_background: wait_background pids 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 130948 130949' 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 130948 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: bash -c 'sleep 1; exit 1' 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/130951: /' 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: bash -c 'sleep 2; exit 0' 2026-02-21T01:40:18.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/130953: /' 2026-02-21T01:40:19.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 1 2026-02-21T01:40:19.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: wait_background: return_code=1 2026-02-21T01:40:19.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-21T01:40:19.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 130949 2026-02-21T01:40:20.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-21T01:40:20.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-21T01:40:20.238 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-21T01:40:20.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 1 2026-02-21T01:40:20.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: test_wait_background: '[' 1 -ne 1 ']' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 130956"' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 130956' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 130957"' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 130957' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: test_wait_background: wait_background pids 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 130956 130957' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 130956 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: bash -c 'sleep 1; exit 0' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: bash -c 'sleep 2; exit 0' 2026-02-21T01:40:20.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/130960: /' 2026-02-21T01:40:20.240 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/130961: /' 2026-02-21T01:40:21.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-21T01:40:21.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-21T01:40:21.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 130957 2026-02-21T01:40:22.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-21T01:40:22.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-21T01:40:22.242 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-21T01:40:22.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-21T01:40:22.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: test_wait_background: '[' 0 -ne 0 ']' 2026-02-21T01:40:22.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: test_wait_background: '[' '!' -z '' ']' 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_for_clean td/ceph-helpers 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: local dir=td/ceph-helpers 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: setup td/ceph-helpers 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:22.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:22.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:22.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:22.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:22.247 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:22.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:22.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:22.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:22.250 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:22.250 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:22.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:22.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:22.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:22.252 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:22.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:22.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:22.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:22.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:22.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:22.255 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:22.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:22.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:22.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:22.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:22.258 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:22.258 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:22.258 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:22.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:40:22.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-21T01:40:22.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:40:22.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:22.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:22.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:22.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:22.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:22.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:22.284 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-21T01:40:22.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:40:22.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:40:22.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:22.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:22.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:40:22.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:40:22.314 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:22.314 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:22.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:22.371 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:22.372 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:22.372 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:22.372 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:22.372 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:22.372 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:22.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:22.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: run_osd td/ceph-helpers 0 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:22.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:22.430 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:22.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:22.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:22.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:40:22.432 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:22.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=78f011f3-b63f-423e-97b3-40cfd6c04128 2026-02-21T01:40:22.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 78f011f3-b63f-423e-97b3-40cfd6c04128' 2026-02-21T01:40:22.433 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 78f011f3-b63f-423e-97b3-40cfd6c04128 2026-02-21T01:40:22.433 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:22.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAGDZlpiX/oGhAAWI2s2WDNEpw6AEmeLFrHmw== 2026-02-21T01:40:22.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAGDZlpiX/oGhAAWI2s2WDNEpw6AEmeLFrHmw=="}' 2026-02-21T01:40:22.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 78f011f3-b63f-423e-97b3-40cfd6c04128 -i td/ceph-helpers/0/new.json 2026-02-21T01:40:22.570 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:22.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:40:22.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGDZlpiX/oGhAAWI2s2WDNEpw6AEmeLFrHmw== --osd-uuid 78f011f3-b63f-423e-97b3-40cfd6c04128 2026-02-21T01:40:22.596 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:22.600+0000 7f5137d9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:22.598 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:22.600+0000 7f5137d9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:22.599 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:22.604+0000 7f5137d9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:22.599 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:22.604+0000 7f5137d9da40 -1 bdev(0x5647cafecc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:22.600 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:22.604+0000 7f5137d9da40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:40:23.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:40:23.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:23.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:40:23.214 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:40:23.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:23.339 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:40:23.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:40:23.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:23.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:23.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:23.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:23.354 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:23.356+0000 7fd3f61e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:23.355 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:23.360+0000 7fd3f61e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:23.356 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:23.360+0000 7fd3f61e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:23.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:40:23.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:23.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:40:23.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:23.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:23.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:23.451 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:23.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:23.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:23.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:23.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:23.719 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:23.724+0000 7fd3f61e8a40 -1 Falling back to public interface 2026-02-21T01:40:24.368 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:24.372+0000 7fd3f61e8a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:40:24.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:24.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:24.566 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:24.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:24.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:24.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:24.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:25.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:25.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:25.678 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:25.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:40:25.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:25.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:25.790 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3862456892,v1:127.0.0.1:6801/3862456892] [v2:127.0.0.1:6802/3862456892,v1:127.0.0.1:6803/3862456892] exists,up 78f011f3-b63f-423e-97b3-40cfd6c04128 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: run_mgr td/ceph-helpers x 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:40:25.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:40:25.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:40:25.793 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:40:25.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:40:25.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:25.905 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:25.906 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:25.906 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:25.906 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:25.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:25.906 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:40:25.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:40:25.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1688: test_wait_for_clean: create_rbd_pool 2026-02-21T01:40:25.925 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:40:26.041 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:40:26.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:40:26.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:40:26.198 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:40:26.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:40:27.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:40:27.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1689: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2026-02-21T01:40:27.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1689: test_wait_for_clean: wait_for_clean 2026-02-21T01:40:27.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:40:27.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 1 .1 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:40:27.487 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:40:27.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2026-02-21T01:40:27.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:40:27.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:40:27.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:40:27.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:40:27.516 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:40:27.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:40:27.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:40:27.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:40:27.631 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:40:27.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=0 2026-02-21T01:40:27.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 0 2026-02-21T01:40:27.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-0' 2026-02-21T01:40:27.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:40:27.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-0 2026-02-21T01:40:27.707 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:40:27.709 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:40:27.710 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-0 2026-02-21T01:40:27.710 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:40:27.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=0 2026-02-21T01:40:27.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 0' 2026-02-21T01:40:27.712 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 0 2026-02-21T01:40:27.712 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:40:27.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 0 -lt 0 2026-02-21T01:40:27.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:40:27.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:27.822 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:27.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 0 == 0 2026-02-21T01:40:27.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: sleep 1 2026-02-21T01:40:28.939 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:40:28.940 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:28.940 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:29.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:29.228 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:29.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:29.443 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:29.443 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:29.443 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:29.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:29.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-21T01:40:29.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-21T01:40:29.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-21T01:40:29.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:29.727 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-21T01:40:29.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:29.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:29.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:29.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:29.829 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:29.829 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:29.829 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:29.829 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:30.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:30.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:30.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:30.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:30.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:30.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:30.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:30.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:30.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:30.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:30.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:30.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:30.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:30.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:30.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:30.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 4 )) 2026-02-21T01:40:30.612 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:30.612 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-21T01:40:30.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:30.811 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:30.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:30.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:30.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:30.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:30.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:30.812 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:31.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:31.027 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:31.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:31.028 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:31.315 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:31.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:31.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:31.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 4 )) 2026-02-21T01:40:31.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:31.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-21T01:40:31.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:31.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:31.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:31.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:31.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:31.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:31.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:31.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:32.212 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:32.212 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:32.212 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:32.213 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:32.500 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:32.501 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:32.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 4 )) 2026-02-21T01:40:32.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:32.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.3 2026-02-21T01:40:33.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:33.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:33.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:33.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:33.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:33.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:33.086 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:33.087 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:33.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:33.293 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:33.293 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:33.294 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:33.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:33.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:33.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:33.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:33.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:33.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 4 >= 4 )) 2026-02-21T01:40:33.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: ceph report 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_fingerprint": "842c990d-6269-4cf1-aa8d-b72f1469f1bc", 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout: "version": "20.2.0-681-g36d388f2", 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout: "commit": "36d388f27052dd940bdbaf543bb05bdba0acc155", 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout: "timestamp": "2026-02-21T01:40:34.131581+0000", 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout: "tag": "", 2026-02-21T01:40:34.126 INFO:tasks.workunit.client.0.trial106.stdout: "health": { 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "status": "HEALTH_WARN", 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "checks": { 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "PG_DEGRADED": { 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "count": 2 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.127 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "TOO_FEW_OSDS": { 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2026-02-21T01:40:34.128 INFO:tasks.workunit.client.0.trial106.stdout: "count": 2 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [], 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "mutes": [] 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "monmap_first_committed": 1, 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "monmap_last_committed": 1, 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "monmap": { 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:40:34.129 INFO:tasks.workunit.client.0.trial106.stdout: "fsid": "640ad1de-49d5-4d28-b121-3f053a140605", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:40:22.281625Z", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "created": "2026-02-21T01:40:22.281625Z", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "min_mon_release": 20, 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "min_mon_release_name": "tentacle", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "election_strategy": 1, 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "disallowed_leaders": "", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode": false, 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "tiebreaker_mon": "", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "removed_ranks": "", 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "features": { 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "persistent": [ 2026-02-21T01:40:34.130 INFO:tasks.workunit.client.0.trial106.stdout: "kraken", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "luminous", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "mimic", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap-prune", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "nautilus", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "octopus", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "pacific", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "elector-pinging", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "quincy", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "reef", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "squid", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "tentacle", 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: "nvmeof_beacon_diff" 2026-02-21T01:40:34.131 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "optional": [] 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "mons": [ 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "rank": 0, 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "name": "a", 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "public_addrs": { 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:7109", 2026-02-21T01:40:34.132 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 0 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:7109/0", 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: "public_addr": "127.0.0.1:7109/0", 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: "priority": 0, 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 0, 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: "crush_location": "{}" 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: "quorum": [ 2026-02-21T01:40:34.133 INFO:tasks.workunit.client.0.trial106.stdout: 0 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap": { 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 12, 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "fsid": "640ad1de-49d5-4d28-b121-3f053a140605", 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "created": "2026-02-21T01:40:22.362255+0000", 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:40:27.485653+0000", 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "last_up_change": "2026-02-21T01:40:24.988862+0000", 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "last_in_change": "2026-02-21T01:40:22.554817+0000", 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "flags_num": 5799936, 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "flags_set": [ 2026-02-21T01:40:34.134 INFO:tasks.workunit.client.0.trial106.stdout: "pglog_hardlimit", 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snapdirs", 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "recovery_deletes", 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "sortbitwise" 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "crush_version": 3, 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "full_ratio": 0.99000000953674316, 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "backfillfull_ratio": 0.99000000953674316, 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "nearfull_ratio": 0.85000002384185791, 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_snapshot": "", 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "pool_max": 1, 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "max_osd": 1, 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "require_min_compat_client": "luminous", 2026-02-21T01:40:34.135 INFO:tasks.workunit.client.0.trial106.stdout: "min_compat_client": "jewel", 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "require_osd_release": "tentacle", 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "allow_crimson": false, 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "pools": [ 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "pool_name": "rbd", 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "create_time": "2026-02-21T01:40:26.155314+0000", 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "flags": 8193, 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "type": 1, 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "size": 2, 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "min_size": 1, 2026-02-21T01:40:34.136 INFO:tasks.workunit.client.0.trial106.stdout: "crush_rule": 0, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_count": 0, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_target": 0, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_barrier": 0, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "is_stretch_pool": false, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "object_hash": 2, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "pg_autoscale_mode": "off", 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num": 4, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "pg_placement_num": 4, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "pg_placement_num_target": 4, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num_target": 4, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num_pending": 4, 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "last_pg_merge_meta": { 2026-02-21T01:40:34.137 INFO:tasks.workunit.client.0.trial106.stdout: "source_pgid": "0.0", 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "ready_epoch": 0, 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_started": 0, 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_clean": 0, 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "source_version": "0'0", 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "target_version": "0'0" 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "last_change": "12", 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend": "0", 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend_prenautilus": "0", 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend_preluminous": "0", 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "auid": 0, 2026-02-21T01:40:34.138 INFO:tasks.workunit.client.0.trial106.stdout: "snap_mode": "selfmanaged", 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "snap_seq": 2, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "snap_epoch": 12, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "pool_snaps": [], 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "removed_snaps": "[]", 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "quota_max_bytes": 0, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "quota_max_objects": 0, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "tiers": [], 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "tier_of": -1, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "read_tier": -1, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "write_tier": -1, 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "cache_mode": "none", 2026-02-21T01:40:34.139 INFO:tasks.workunit.client.0.trial106.stdout: "target_max_bytes": 0, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "target_max_objects": 0, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_dirty_ratio_micro": 400000, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_full_ratio_micro": 800000, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "cache_min_flush_age": 0, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "cache_min_evict_age": 0, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "erasure_code_profile": "", 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_params": { 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "type": "none" 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_period": 0, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_count": 0, 2026-02-21T01:40:34.140 INFO:tasks.workunit.client.0.trial106.stdout: "use_gmt_hitset": true, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "min_read_recency_for_promote": 0, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "min_write_recency_for_promote": 0, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_grade_decay_rate": 0, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_search_last_n": 0, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "grade_table": [], 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "stripe_width": 0, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "expected_num_objects": 0, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "fast_read": false, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "nonprimary_shards": "{}", 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "options": {}, 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "application_metadata": { 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: "rbd": {} 2026-02-21T01:40:34.141 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "read_balance": { 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "score_type": "Fair distribution", 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "score_acting": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "score_stable": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "optimal_score": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "raw_score_acting": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "raw_score_stable": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "primary_affinity_weighted": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "average_primary_affinity": 1, 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: "average_primary_affinity_weighted": 1 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.142 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "osds": [ 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "uuid": "78f011f3-b63f-423e-97b3-40cfd6c04128", 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "up": 1, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "in": 1, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 1, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "primary_affinity": 1, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "last_clean_begin": 0, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "last_clean_end": 0, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 5, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "up_thru": 7, 2026-02-21T01:40:34.143 INFO:tasks.workunit.client.0.trial106.stdout: "down_at": 0, 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "lost_at": 0, 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "public_addrs": { 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6800", 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6801", 2026-02-21T01:40:34.144 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_addrs": { 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6802", 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6803", 2026-02-21T01:40:34.145 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_back_addrs": { 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6806", 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6807", 2026-02-21T01:40:34.146 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_front_addrs": { 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6804", 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6805", 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 3862456892 2026-02-21T01:40:34.147 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "public_addr": "127.0.0.1:6801/3862456892", 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_addr": "127.0.0.1:6803/3862456892", 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_back_addr": "127.0.0.1:6807/3862456892", 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_front_addr": "127.0.0.1:6805/3862456892", 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "state": [ 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "exists", 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "up" 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "osd_xinfo": [ 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.148 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "down_stamp": "0.000000", 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "laggy_probability": 0, 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "laggy_interval": 0, 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "features": 4544132024016699391, 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "old_weight": 0, 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "last_purged_snaps_scrub": "2026-02-21T01:40:25.398409+0000", 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "dead_epoch": 0 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap": [], 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap_items": [], 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap_primaries": [], 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "pg_temp": [], 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "primary_temp": [], 2026-02-21T01:40:34.149 INFO:tasks.workunit.client.0.trial106.stdout: "blocklist": {}, 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "range_blocklist": {}, 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "erasure_code_profiles": { 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "default": { 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "k": "2", 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "m": "2", 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "plugin": "isa", 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "technique": "reed_sol_van" 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "removed_snaps_queue": [ 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "snaps": [ 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "begin": 2, 2026-02-21T01:40:34.150 INFO:tasks.workunit.client.0.trial106.stdout: "length": 1 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: "new_removed_snaps": [ 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: "snaps": [ 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: "begin": 2, 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: "length": 1 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.151 INFO:tasks.workunit.client.0.trial106.stdout: "new_purged_snaps": [], 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "crush_node_flags": {}, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "device_class_flags": {}, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode": { 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode_enabled": false, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_bucket_count": 0, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "degraded_stretch_mode": 0, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "recovering_stretch_mode": 0, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode_bucket": 0 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "osd_metadata": [ 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "arch": "x86_64", 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "back_addr": "[v2:127.0.0.1:6802/3862456892,v1:127.0.0.1:6803/3862456892]", 2026-02-21T01:40:34.152 INFO:tasks.workunit.client.0.trial106.stdout: "back_iface": "", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs": "1", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_dedicated_db": "0", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_dedicated_wal": "0", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_single_shared_device": "1", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_allocation_from_file": "1", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_allocator": "hybrid", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_access_mode": "file", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_block_size": "4096", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_devices": "nvme0n1", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_driver": "KernelDevice", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_optimal_io_size": "4096", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_rotational": "0", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_size": "107374182400", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_support_discard": "1", 2026-02-21T01:40:34.153 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_type": "ssd", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_min_alloc_size": "4096", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_onode_segmentation": "inactive", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_write_mode": "classic", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_release": "tentacle", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version": "ceph version 20.2.0-681-g36d388f2 (36d388f27052dd940bdbaf543bb05bdba0acc155) tentacle (stable - None)", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version_short": "20.2.0-681-g36d388f2", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version_when_created": "ceph version 20.2.0-681-g36d388f2 (36d388f27052dd940bdbaf543bb05bdba0acc155) tentacle (stable - None)", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "cpu": "AMD EPYC 4464P 12-Core Processor", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "created_at": "2026-02-21T01:40:23.124101Z", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "default_device_class": "ssd", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "device_ids": "nvme0n1=Micron_7450_MTFDKBA960TFR_25174FCD6E72", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "devices": "nvme0n1", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "distro": "ubuntu", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "distro_description": "Ubuntu 22.04.5 LTS", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "distro_version": "22.04", 2026-02-21T01:40:34.154 INFO:tasks.workunit.client.0.trial106.stdout: "front_addr": "[v2:127.0.0.1:6800/3862456892,v1:127.0.0.1:6801/3862456892]", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "front_iface": "", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/3862456892,v1:127.0.0.1:6807/3862456892]", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/3862456892,v1:127.0.0.1:6805/3862456892]", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "hostname": "trial106", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "journal_rotational": "0", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "kernel_description": "#180-Ubuntu SMP Fri Jan 9 16:10:31 UTC 2026", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "kernel_version": "5.15.0-170-generic", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "mem_swap_kb": "8388604", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "mem_total_kb": "130961696", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "objectstore_numa_node": "0", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "objectstore_numa_nodes": "0", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "os": "Linux", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "osd_data": "td/ceph-helpers/0", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "osd_objectstore": "bluestore", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "osdspec_affinity": "", 2026-02-21T01:40:34.155 INFO:tasks.workunit.client.0.trial106.stdout: "rotational": "0" 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_clean_epochs": { 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "min_last_epoch_clean": 0, 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_clean": { 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "per_pool": [] 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "osd_epochs": [ 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 5 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_first_committed": 1, 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_last_committed": 12, 2026-02-21T01:40:34.156 INFO:tasks.workunit.client.0.trial106.stdout: "crushmap": { 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "devices": [ 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "name": "osd.0", 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "class": "ssd" 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "types": [ 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 0, 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "name": "osd" 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "name": "host" 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 2, 2026-02-21T01:40:34.157 INFO:tasks.workunit.client.0.trial106.stdout: "name": "chassis" 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 3, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "name": "rack" 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 4, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "name": "row" 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 5, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "name": "pdu" 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 6, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "name": "pod" 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 7, 2026-02-21T01:40:34.158 INFO:tasks.workunit.client.0.trial106.stdout: "name": "room" 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 8, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "name": "datacenter" 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 9, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "name": "zone" 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 10, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "name": "region" 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "name": "root" 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.159 INFO:tasks.workunit.client.0.trial106.stdout: "buckets": [ 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "id": -1, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "name": "default", 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "root", 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "id": -3, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "id": -2, 2026-02-21T01:40:34.160 INFO:tasks.workunit.client.0.trial106.stdout: "name": "default~ssd", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "root", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "id": -4, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "id": -3, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "name": "trial106", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "host", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:40:34.161 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "id": -4, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "name": "trial106~ssd", 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "host", 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:40:34.162 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "rules": [ 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "rule_id": 0, 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "rule_name": "replicated_rule", 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "type": 1, 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "steps": [ 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "op": "take", 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "item": -1, 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "item_name": "default" 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "op": "choose_firstn", 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "num": 0, 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "type": "osd" 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.163 INFO:tasks.workunit.client.0.trial106.stdout: "op": "emit" 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "tunables": { 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "choose_local_tries": 0, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "choose_local_fallback_tries": 0, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "choose_total_tries": 50, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_descend_once": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_vary_r": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_stable": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "msr_descents": 100, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "msr_collision_tries": 100, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "straw_calc_version": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "allowed_bucket_algs": 54, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "profile": "jewel", 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "optimal_tunables": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "legacy_tunables": 0, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "minimum_required_version": "jewel", 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables2": 1, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "has_v2_rules": 0, 2026-02-21T01:40:34.164 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables3": 1, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "has_v3_rules": 0, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "has_v4_buckets": 1, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables5": 1, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "has_v5_rules": 0, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "has_msr_rules": 0 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "choose_args": {} 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "fsmap": { 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "btime": "2026-02-21T01:40:22:362042+0000", 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "default_fscid": -1, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "compat": { 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "compat": {}, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "ro_compat": {}, 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "incompat": { 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "feature_1": "base v0.20", 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "feature_2": "client writeable ranges", 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "feature_3": "default file layouts on dirs", 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "feature_4": "dir inode in separate object", 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "feature_5": "mds uses versioned encoding", 2026-02-21T01:40:34.165 INFO:tasks.workunit.client.0.trial106.stdout: "feature_6": "dirfrag is stored in omap", 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "feature_8": "no anchor table", 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "feature_9": "file layout v2", 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "feature_10": "snaprealm v2", 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "feature_11": "minor log segments", 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "feature_12": "quiesce subvolumes" 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "feature_flags": { 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "enable_multiple": true, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "ever_enabled_multiple": true 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "standbys": [], 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "filesystems": [] 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "mdsmap_first_committed": 1, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "mdsmap_last_committed": 1, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "auth": { 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "first_committed": 1, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "last_committed": 3, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "num_secrets": 1 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.166 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg": 4, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_active": 4, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_unknown": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_osd": 1, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "pool_sum": { 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "stat_sum": { 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes": 19, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects": 2, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_clones": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_copies": 4, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing_on_primary": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_degraded": 2, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_misplaced": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_unfound": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_dirty": 2, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_whiteouts": 0, 2026-02-21T01:40:34.167 INFO:tasks.workunit.client.0.trial106.stdout: "num_read": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_read_kb": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_write": 2, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_write_kb": 2, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_scrub_errors": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_shallow_scrub_errors": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_deep_scrub_errors": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_recovered": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_recovered": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_keys_recovered": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_omap": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_hit_set_archive": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_hit_set_archive": 0, 2026-02-21T01:40:34.168 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_kb": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_kb": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_promote": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_high": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_low": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_some": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_full": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_pinned": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_legacy_snapsets": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_large_omap_objects": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_manifest": 0, 2026-02-21T01:40:34.169 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_bytes": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_keys": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_repaired": 0 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "store_stats": { 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "total": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "available": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:40:34.170 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 0, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 0 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "log_size": 3, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "ondisk_log_size": 3, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "up": 4, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "acting": 4, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "num_store_stats": 0 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "osd_sum": { 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 0, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 0, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "num_pgs": 0, 2026-02-21T01:40:34.171 INFO:tasks.workunit.client.0.trial106.stdout: "num_osds": 1, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_osds": 1, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_omap_osds": 1, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "kb": 104857600, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used": 27060, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_data": 100, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_omap": 7, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_meta": 26936, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "kb_avail": 104830540, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "statfs": { 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "total": 107374182400, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "available": 107346472960, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:40:34.172 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 102400, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 20595, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 7483, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 27583173 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "hb_peers": [], 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "snap_trim_queue_len": 0, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "num_snap_trimming": 0, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "num_shards_repaired": 0, 2026-02-21T01:40:34.173 INFO:tasks.workunit.client.0.trial106.stdout: "op_queue_age_hist": { 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "histogram": [], 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "upper_bound": 1 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "perf_stat": { 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ms": 0, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ms": 0, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ns": 0, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ns": 0 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "alerts": [], 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "network_ping_times": [] 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "osd_sum_by_class": { 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "ssd": { 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 0, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 0, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "num_pgs": 0, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "num_osds": 1, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_osds": 1, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_omap_osds": 1, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "kb": 104857600, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used": 27060, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_data": 100, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_omap": 7, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_meta": 26936, 2026-02-21T01:40:34.174 INFO:tasks.workunit.client.0.trial106.stdout: "kb_avail": 104830540, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "statfs": { 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "total": 107374182400, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "available": 107346472960, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 102400, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 20595, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 7483, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 27583173 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "hb_peers": [], 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "snap_trim_queue_len": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "num_snap_trimming": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "num_shards_repaired": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "op_queue_age_hist": { 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "histogram": [], 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "upper_bound": 1 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "perf_stat": { 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ms": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ms": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ns": 0, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ns": 0 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "alerts": [], 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: "network_ping_times": [] 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.175 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "pool_stats": [ 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "poolid": 1, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg": 4, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "stat_sum": { 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes": 19, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects": 2, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_clones": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_copies": 4, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing_on_primary": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_degraded": 2, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_misplaced": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_unfound": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_dirty": 2, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_whiteouts": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_read": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_read_kb": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_write": 2, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_write_kb": 2, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_scrub_errors": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_shallow_scrub_errors": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_deep_scrub_errors": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_recovered": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_recovered": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_keys_recovered": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_omap": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_hit_set_archive": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_hit_set_archive": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_kb": 0, 2026-02-21T01:40:34.176 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_kb": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_promote": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_high": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_low": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_some": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_full": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_pinned": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_legacy_snapsets": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_large_omap_objects": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_manifest": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_bytes": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_keys": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_repaired": 0 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "store_stats": { 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "total": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "available": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 4096, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 19, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 0, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 0 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "log_size": 3, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "ondisk_log_size": 3, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "up": 4, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "acting": 4, 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: "num_store_stats": 1 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.177 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "osd_stats": [ 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 21474836481 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_by_state": [ 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "state": "active+undersized+degraded", 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num": 2 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "state": "active+undersized", 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num": 2 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "pool_pg_unavailable_map": [ 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "poolid": "1", 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "pgs": [] 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_by_osd": [ 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num_primary_pg": 4, 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num_acting_pg": 4, 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "num_up_not_acting_pg": 0 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snaps": [ 2026-02-21T01:40:34.178 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snaps": {} 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "servicemap": { 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:40:22.362444+0000", 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "services": {} 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "mgrstat_first_committed": 1, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "mgrstat_last_committed": 4, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "logm_first_committed": 1, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "logm_last_committed": 56, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "paxos": { 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "first_committed": 1, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "last_committed": 81, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "last_pn": 0, 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: "accepted_pn": 0 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:40:34.179 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:40:34.180 INFO:tasks.workunit.client.0.trial106.stderr:report 3304614278 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 1 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1690: test_wait_for_clean: run_osd td/ceph-helpers 1 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:34.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=667a0dfd-0275-4301-8f20-e8dc9d34c0b1 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 667a0dfd-0275-4301-8f20-e8dc9d34c0b1' 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 667a0dfd-0275-4301-8f20-e8dc9d34c0b1 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASDZlp/4ttCRAAmt0P9AWoMIMsL63bcdeZJQ== 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASDZlp/4ttCRAAmt0P9AWoMIMsL63bcdeZJQ=="}' 2026-02-21T01:40:34.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 667a0dfd-0275-4301-8f20-e8dc9d34c0b1 -i td/ceph-helpers/1/new.json 2026-02-21T01:40:34.367 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:34.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:40:34.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASDZlp/4ttCRAAmt0P9AWoMIMsL63bcdeZJQ== --osd-uuid 667a0dfd-0275-4301-8f20-e8dc9d34c0b1 2026-02-21T01:40:34.403 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:34.408+0000 7f0e23987a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:34.405 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:34.408+0000 7f0e23987a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:34.406 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:34.408+0000 7f0e23987a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:34.406 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:34.412+0000 7f0e23987a40 -1 bdev(0x55a21682ec00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:34.406 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:34.412+0000 7f0e23987a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:40:35.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:40:35.049 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:35.050 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:40:35.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:40:35.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:35.347 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:40:35.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:40:35.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:35.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:35.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:35.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:35.364 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:35.368+0000 7fbca827ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:35.365 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:35.368+0000 7fbca827ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:35.366 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:35.368+0000 7fbca827ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:35.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:35.556 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:35.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:35.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:35.732 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:35.736+0000 7fbca827ca40 -1 Falling back to public interface 2026-02-21T01:40:35.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:36.343 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:36.348+0000 7fbca827ca40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:40:36.775 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:36.775 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:36.775 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:36.775 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:36.775 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:36.775 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:36.990 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/241303761,v1:127.0.0.1:6811/241303761] [v2:127.0.0.1:6812/241303761,v1:127.0.0.1:6813/241303761] exists,up 667a0dfd-0275-4301-8f20-e8dc9d34c0b1 2026-02-21T01:40:36.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:36.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:36.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:36.990 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1691: test_wait_for_clean: wait_for_clean 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:40:36.991 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:40:37.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:40:37.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:40:37.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:40:37.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:40:37.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:40:37.097 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:40:37.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:40:37.307 INFO:tasks.workunit.client.0.trial106.stderr:1' 2026-02-21T01:40:37.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:40:37.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:40:37.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:40:37.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-21T01:40:37.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-21T01:40:37.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T01:40:37.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:40:37.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:40:37.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=0 2026-02-21T01:40:37.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 0 2026-02-21T01:40:37.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-0' 2026-02-21T01:40:37.468 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:40:37.468 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-21T01:40:37.468 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:40:37.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:40:37.470 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-21T01:40:37.470 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:40:37.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-21T01:40:37.471 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836484 2026-02-21T01:40:37.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T01:40:37.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:40:37.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-21T01:40:37.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:40:38.702 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:40:38.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:40:38.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-21T01:40:38.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:40:38.908 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-0 2026-02-21T01:40:38.909 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:40:38.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:40:38.910 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-0 2026-02-21T01:40:38.910 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:40:38.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=0 2026-02-21T01:40:38.912 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 0 2026-02-21T01:40:38.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 0' 2026-02-21T01:40:38.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:40:39.130 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 68719476737 -lt 0 2026-02-21T01:40:39.130 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:40:39.131 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:39.131 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:39.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:40:39.415 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:39.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:39.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:39.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:39.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:39.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:39.416 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:39.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:39.618 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:39.619 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:39.619 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:39.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:39.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-21T01:40:39.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-21T01:40:39.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-21T01:40:39.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:39.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-21T01:40:39.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:39.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:39.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:39.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:39.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:39.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:39.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:39.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:40.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:40.198 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:40.198 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:40.198 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:40.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:40.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:40.476 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:40.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:40.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:40.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-21T01:40:40.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:40.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-21T01:40:40.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:40.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:40.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:40.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:40.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:40.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:40.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:40.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:41.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:41.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:41.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:41.167 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:41.445 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:41.446 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:41.446 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:41.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:41.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:41.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-21T01:40:41.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:41.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-21T01:40:42.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:42.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:42.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:42.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:42.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:42.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:42.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:42.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:42.320 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:42.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:42.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:42.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:42.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:42.595 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:42.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:42.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:42.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:42.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:42.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:42.596 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:42.597 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:42.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:40:42.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:40:42.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-21T01:40:42.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:42.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-21T01:40:43.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:43.667 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:43.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:43.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:43.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:43.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:43.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:43.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:43.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:40:43.881 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:43.881 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:43.882 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:40:44.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:40:44.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:40:44.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:40:44.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2026-02-21T01:40:44.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2026-02-21T01:40:44.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1668: wait_for_clean: loop=0 2026-02-21T01:40:44.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:40:44.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-21T01:40:44.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:40:44.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:40:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:40:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:40:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:40:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:40:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:40:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:40:44.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:40:44.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:40:44.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:40:44.746 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1692: test_wait_for_clean: teardown td/ceph-helpers 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:45.029 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:45.030 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:45.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:45.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:45.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:45.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:45.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:45.139 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:45.147 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:45.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:45.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_for_health_ok td/ceph-helpers 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1854: test_wait_for_health_ok: local dir=td/ceph-helpers 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_wait_for_health_ok: setup td/ceph-helpers 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:40:45.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:40:45.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:45.156 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:45.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:45.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:45.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:45.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:45.158 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:40:45.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:40:45.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:40:45.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:40:45.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:40:45.161 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:40:45.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:45.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:40:45.162 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:40:45.163 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:40:45.164 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:40:45.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:40:45.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:40:45.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:40:45.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:40:45.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:40:45.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:40:45.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:40:45.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:40:45.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:40:45.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:40:45.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:40:45.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:40:45.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:40:45.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2026-02-21T01:40:45.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:40:45.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:45.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:45.194 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:45.194 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.194 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:45.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2026-02-21T01:40:45.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:45.225 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:45.226 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:45.226 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.226 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.226 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:45.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:45.226 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:40:45.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:40:45.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:40:45.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:40:45.281 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:40:45.282 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:40:45.282 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:40:45.282 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:40:45.282 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:40:45.283 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:40:45.283 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.283 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.283 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:40:45.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:40:45.283 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1858: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:40:45.337 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:40:45.446 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:40:45.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:45.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:45.447 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:45.447 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.447 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.447 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:45.448 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:40:45.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1860: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:45.470 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:45.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:40:45.472 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:45.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=51211fef-d469-4043-86c2-94dbd0a24791 2026-02-21T01:40:45.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 51211fef-d469-4043-86c2-94dbd0a24791' 2026-02-21T01:40:45.473 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 51211fef-d469-4043-86c2-94dbd0a24791 2026-02-21T01:40:45.473 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:45.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdDZlpY9Q2HRAAUGmgK2EyBkLOrMj+Lxs06Q== 2026-02-21T01:40:45.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdDZlpY9Q2HRAAUGmgK2EyBkLOrMj+Lxs06Q=="}' 2026-02-21T01:40:45.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 51211fef-d469-4043-86c2-94dbd0a24791 -i td/ceph-helpers/0/new.json 2026-02-21T01:40:45.600 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:45.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:40:45.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdDZlpY9Q2HRAAUGmgK2EyBkLOrMj+Lxs06Q== --osd-uuid 51211fef-d469-4043-86c2-94dbd0a24791 2026-02-21T01:40:45.625 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:45.627+0000 7f47c0c7ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:45.627 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:45.631+0000 7f47c0c7ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:45.627 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:45.631+0000 7f47c0c7ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:45.628 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:45.631+0000 7f47c0c7ca40 -1 bdev(0x557757078c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:45.628 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:45.631+0000 7f47c0c7ca40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:40:46.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:40:46.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:46.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:40:46.379 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:40:46.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:46.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:40:46.498 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:40:46.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:46.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:46.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:46.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:46.523 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:46.527+0000 7f5b1607da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:46.525 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:46.527+0000 7f5b1607da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:46.526 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:46.531+0000 7f5b1607da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:46.604 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:46.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:46.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:46.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:46.709 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:47.028 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:47.031+0000 7f5b1607da40 -1 Falling back to public interface 2026-02-21T01:40:47.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:47.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:47.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:47.710 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:47.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:47.711 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:47.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:48.000 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:48.003+0000 7f5b1607da40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:40:48.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:48.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:48.820 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:48.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:40:48.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:48.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:40:49.029 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/79526462,v1:127.0.0.1:6801/79526462] [v2:127.0.0.1:6802/79526462,v1:127.0.0.1:6803/79526462] exists,up 51211fef-d469-4043-86c2-94dbd0a24791 2026-02-21T01:40:49.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:49.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:49.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1861: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:49.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:49.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=965ec3b0-4d45-4b7d-9330-25c277cb7c23 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 965ec3b0-4d45-4b7d-9330-25c277cb7c23 2026-02-21T01:40:49.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 965ec3b0-4d45-4b7d-9330-25c277cb7c23' 2026-02-21T01:40:49.034 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:49.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhDZlpVUUsAxAA3bA2Vn2Tbwr3KZzE378OyQ== 2026-02-21T01:40:49.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhDZlpVUUsAxAA3bA2Vn2Tbwr3KZzE378OyQ=="}' 2026-02-21T01:40:49.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 965ec3b0-4d45-4b7d-9330-25c277cb7c23 -i td/ceph-helpers/1/new.json 2026-02-21T01:40:49.261 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:49.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:40:49.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhDZlpVUUsAxAA3bA2Vn2Tbwr3KZzE378OyQ== --osd-uuid 965ec3b0-4d45-4b7d-9330-25c277cb7c23 2026-02-21T01:40:49.296 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:49.299+0000 7f552259ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:49.298 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:49.299+0000 7f552259ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:49.299 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:49.303+0000 7f552259ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:49.299 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:49.303+0000 7f552259ca40 -1 bdev(0x55e6010b4c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:49.300 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:49.303+0000 7f552259ca40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:40:49.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:40:49.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:49.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:40:49.895 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:40:49.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:50.194 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:40:50.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:40:50.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:50.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:50.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:50.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:50.209 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:50.211+0000 7fbc5cd25a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:50.210 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:50.215+0000 7fbc5cd25a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:50.211 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:50.215+0000 7fbc5cd25a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:40:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:40:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:50.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:50.402 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:50.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:50.632 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:50.635+0000 7fbc5cd25a40 -1 Falling back to public interface 2026-02-21T01:40:51.265 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:51.267+0000 7fbc5cd25a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:40:51.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:51.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:51.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:51.616 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:51.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:51.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:51.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:52.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:52.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:52.830 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:52.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:40:52.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:52.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:40:53.043 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2949481937,v1:127.0.0.1:6811/2949481937] [v2:127.0.0.1:6812/2949481937,v1:127.0.0.1:6813/2949481937] exists,up 965ec3b0-4d45-4b7d-9330-25c277cb7c23 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1862: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/2 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:40:53.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:40:53.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:40:53.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/2 2026-02-21T01:40:53.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:40:53.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=11a64f1a-b632-404c-9501-6cb644049ba1 2026-02-21T01:40:53.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 11a64f1a-b632-404c-9501-6cb644049ba1' 2026-02-21T01:40:53.048 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 11a64f1a-b632-404c-9501-6cb644049ba1 2026-02-21T01:40:53.048 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:40:53.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAlDZlpSSf6AxAAEslSLHW1umI7mda0cfa5ug== 2026-02-21T01:40:53.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAlDZlpSSf6AxAAEslSLHW1umI7mda0cfa5ug=="}' 2026-02-21T01:40:53.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 11a64f1a-b632-404c-9501-6cb644049ba1 -i td/ceph-helpers/2/new.json 2026-02-21T01:40:53.270 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:53.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/2/new.json 2026-02-21T01:40:53.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlDZlpSSf6AxAAEslSLHW1umI7mda0cfa5ug== --osd-uuid 11a64f1a-b632-404c-9501-6cb644049ba1 2026-02-21T01:40:53.297 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:53.299+0000 7f48cff01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:53.298 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:53.299+0000 7f48cff01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:53.299 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:53.303+0000 7f48cff01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:53.299 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:53.303+0000 7f48cff01a40 -1 bdev(0x5592c7308c00 td/ceph-helpers/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:40:53.300 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:53.303+0000 7f48cff01a40 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2026-02-21T01:40:53.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/2/keyring 2026-02-21T01:40:53.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:40:53.930 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:40:53.931 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:40:53.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:40:54.209 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:40:54.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:40:54.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:40:54.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:40:54.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:40:54.210 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:40:54.234 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:54.239+0000 7fb9cb167a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:54.236 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:54.239+0000 7fb9cb167a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:54.237 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:54.239+0000 7fb9cb167a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:40:54.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:40:54.427 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:40:54.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:54.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:40:54.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:54.656 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:54.659+0000 7fb9cb167a40 -1 Falling back to public interface 2026-02-21T01:40:55.269 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:40:55.271+0000 7fb9cb167a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:40:55.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:55.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:55.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:40:55.635 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:40:55.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:55.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:40:55.854 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:40:56.856 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:40:56.856 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:40:56.857 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:40:56.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:40:56.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:40:56.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:40:57.061 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3199102918,v1:127.0.0.1:6819/3199102918] [v2:127.0.0.1:6820/3199102918,v1:127.0.0.1:6821/3199102918] exists,up 11a64f1a-b632-404c-9501-6cb644049ba1 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1863: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:40:57.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:40:57.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:40:57.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1864: test_wait_for_health_ok: ceph osd down 0 2026-02-21T01:40:57.589 INFO:tasks.workunit.client.0.trial106.stderr:osd.0 is already down. 2026-02-21T01:40:57.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_wait_for_health_ok: TIMEOUT=1 2026-02-21T01:40:57.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1866: test_wait_for_health_ok: wait_for_health_ok 2026-02-21T01:40:57.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: wait_for_health_ok: wait_for_health HEALTH_OK 2026-02-21T01:40:57.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: wait_for_health: local grepstr=HEALTH_OK 2026-02-21T01:40:57.600 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: get_timeout_delays 1 .1 2026-02-21T01:40:57.600 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:40:57.600 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:40:57.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:40:57.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:40:57.601 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:40:57.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2026-02-21T01:40:57.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: local -a delays 2026-02-21T01:40:57.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: wait_for_health: local -i loop=0 2026-02-21T01:40:57.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-21T01:40:57.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep HEALTH_OK 2026-02-21T01:40:57.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: wait_for_health: (( 0 >= 4 )) 2026-02-21T01:40:57.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: wait_for_health: sleep 0.1 2026-02-21T01:40:58.004 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: loop+=1 2026-02-21T01:40:58.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-21T01:40:58.005 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep HEALTH_OK 2026-02-21T01:40:58.281 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: wait_for_health: (( 1 >= 4 )) 2026-02-21T01:40:58.281 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: wait_for_health: sleep 0.2 2026-02-21T01:40:58.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: loop+=1 2026-02-21T01:40:58.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-21T01:40:58.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep HEALTH_OK 2026-02-21T01:40:58.758 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: wait_for_health: (( 2 >= 4 )) 2026-02-21T01:40:58.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: wait_for_health: sleep 0.4 2026-02-21T01:40:59.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: loop+=1 2026-02-21T01:40:59.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-21T01:40:59.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep HEALTH_OK 2026-02-21T01:40:59.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: wait_for_health: (( 3 >= 4 )) 2026-02-21T01:40:59.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: wait_for_health: sleep 0.3 2026-02-21T01:40:59.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: wait_for_health: loop+=1 2026-02-21T01:40:59.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-21T01:40:59.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep HEALTH_OK 2026-02-21T01:41:00.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: wait_for_health: (( 4 >= 4 )) 2026-02-21T01:41:00.007 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: wait_for_health: ceph health detail 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] OSD_DOWN: 3 osds down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout: osd.0 (root=default,host=trial106) is down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout: osd.1 (root=default,host=trial106) is down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout: osd.2 (root=default,host=trial106) is down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout: host trial106 (root=default) (3 osds) is down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2026-02-21T01:41:00.291 INFO:tasks.workunit.client.0.trial106.stdout: root default (3 osds) is down 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: wait_for_health: return 1 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: wait_for_health_ok: return 1 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1868: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:41:00.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:00.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:41:00.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:41:00.304 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:41:00.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:41:00.304 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:00.304 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T01:41:00.305 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:41:00.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:41:00.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:41:00.306 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:41:00.318 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:00.323+0000 7f72239f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:00.320 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:00.323+0000 7f72239f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:00.321 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:00.323+0000 7f72239f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:00.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:00.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:00.737 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:00.800 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:00.803+0000 7f72239f8a40 -1 Falling back to public interface 2026-02-21T01:41:01.380 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:01.383+0000 7f72239f8a40 -1 osd.0 14 log_to_monitors true 2026-02-21T01:41:01.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:01.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:01.739 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:01.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:01.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:01.739 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:01.953 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 18 up_thru 0 down_at 16 last_clean_interval [5,14) [v2:127.0.0.1:6800/923911429,v1:127.0.0.1:6801/923911429] [v2:127.0.0.1:6802/923911429,v1:127.0.0.1:6803/923911429] exists,up 51211fef-d469-4043-86c2-94dbd0a24791 2026-02-21T01:41:01.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:01.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:01.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1869: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:01.957 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:01.958 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:01.959 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:41:01.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:41:01.961 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/1/whoami 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:41:01.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:41:01.980 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:01.983+0000 7f55ef128a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:01.981 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:01.983+0000 7f55ef128a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:01.982 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:01.983+0000 7f55ef128a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:02.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:02.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:02.380 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:02.383+0000 7f55ef128a40 -1 Falling back to public interface 2026-02-21T01:41:02.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:02.997 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:02.999+0000 7f55ef128a40 -1 osd.1 14 log_to_monitors true 2026-02-21T01:41:03.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:03.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:03.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:03.391 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:03.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:03.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 20 up_thru 0 down_at 16 last_clean_interval [10,14) [v2:127.0.0.1:6810/316951706,v1:127.0.0.1:6811/316951706] [v2:127.0.0.1:6812/316951706,v1:127.0.0.1:6813/316951706] exists,up 965ec3b0-4d45-4b7d-9330-25c277cb7c23 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1870: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-21T01:41:03.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:41:03.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/2 2026-02-21T01:41:03.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:03.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:03.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:41:03.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:41:03.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/2 2026-02-21T01:41:03.609 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:41:03.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-21T01:41:03.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:03.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/2/whoami 2026-02-21T01:41:03.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-21T01:41:03.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:41:03.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:41:03.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:41:03.627 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:03.631+0000 7fa7d3a6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:03.629 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:03.631+0000 7fa7d3a6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:03.630 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:03.631+0000 7fa7d3a6fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:03.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:41:04.020 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:04.023+0000 7fa7d3a6fa40 -1 Falling back to public interface 2026-02-21T01:41:04.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:04.630 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:04.630+0000 7fa7d3a6fa40 -1 osd.2 15 log_to_monitors true 2026-02-21T01:41:05.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:05.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:05.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:05.028 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:05.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:05.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 16 last_clean_interval [14,15) [v2:127.0.0.1:6818/3610080407,v1:127.0.0.1:6819/3610080407] [v2:127.0.0.1:6820/3610080407,v1:127.0.0.1:6821/3610080407] exists,up 11a64f1a-b632-404c-9501-6cb644049ba1 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1871: test_wait_for_health_ok: wait_for_health_ok 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: wait_for_health_ok: wait_for_health HEALTH_OK 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: wait_for_health: local grepstr=HEALTH_OK 2026-02-21T01:41:05.244 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: get_timeout_delays 300 .1 2026-02-21T01:41:05.245 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:41:05.245 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:41:05.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:41:05.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:41:05.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:41:05.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2026-02-21T01:41:05.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: local -a delays 2026-02-21T01:41:05.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: wait_for_health: local -i loop=0 2026-02-21T01:41:05.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-21T01:41:05.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep HEALTH_OK 2026-02-21T01:41:05.748 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_OK 2026-02-21T01:41:05.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: test_wait_for_health_ok: teardown td/ceph-helpers 2026-02-21T01:41:05.749 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:05.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:06.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:06.060 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:06.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:06.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:06.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:06.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:06.064 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:06.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:06.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:06.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:06.066 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:06.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:06.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:06.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:06.076 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:06.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.077 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:06.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:06.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:06.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_for_osd td/ceph-helpers 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:986: test_wait_for_osd: local dir=td/ceph-helpers 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: test_wait_for_osd: setup td/ceph-helpers 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:06.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:06.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:06.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:06.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:06.082 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:06.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:06.083 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:06.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:06.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:06.085 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:06.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:06.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:06.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:06.087 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:06.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:06.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:06.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:06.090 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:06.090 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.090 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:06.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:06.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:06.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:41:06.092 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:41:06.092 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.093 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:41:06.094 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:41:06.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:41:06.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:41:06.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:06.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:06.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:41:06.150 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:41:06.150 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:06.150 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:06.150 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:41:06.150 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:06.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:41:06.207 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.208 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:06.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:06.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:41:06.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:989: test_wait_for_osd: run_mgr td/ceph-helpers x 2026-02-21T01:41:06.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:41:06.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:41:06.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:41:06.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:41:06.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:41:06.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:06.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:06.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:990: test_wait_for_osd: run_osd td/ceph-helpers 0 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:06.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:06.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:06.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:06.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:06.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:41:06.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:41:06.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:06.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=02c72d3a-8ffd-4deb-bce7-879dcb50aa41 2026-02-21T01:41:06.396 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 02c72d3a-8ffd-4deb-bce7-879dcb50aa41 2026-02-21T01:41:06.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 02c72d3a-8ffd-4deb-bce7-879dcb50aa41' 2026-02-21T01:41:06.396 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:06.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAyDZlproCqGBAAsdFBwQ7fHPbJQiyu38r8yw== 2026-02-21T01:41:06.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAyDZlproCqGBAAsdFBwQ7fHPbJQiyu38r8yw=="}' 2026-02-21T01:41:06.408 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 02c72d3a-8ffd-4deb-bce7-879dcb50aa41 -i td/ceph-helpers/0/new.json 2026-02-21T01:41:06.523 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:06.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:41:06.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyDZlproCqGBAAsdFBwQ7fHPbJQiyu38r8yw== --osd-uuid 02c72d3a-8ffd-4deb-bce7-879dcb50aa41 2026-02-21T01:41:06.549 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:06.550+0000 7f44f1e12a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:06.550 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:06.550+0000 7f44f1e12a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:06.551 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:06.554+0000 7f44f1e12a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:06.551 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:06.554+0000 7f44f1e12a40 -1 bdev(0x55a5f7d34c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:06.551 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:06.554+0000 7f44f1e12a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:41:07.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:41:07.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:07.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:41:07.311 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:41:07.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:07.436 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:41:07.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:41:07.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:07.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:07.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:07.437 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:07.461 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:07.462+0000 7f7827a59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:07.463 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:07.466+0000 7f7827a59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:07.464 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:07.466+0000 7f7827a59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:07.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:41:07.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:07.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:07.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:07.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:07.546 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:07.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:07.547 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:07.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:07.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:07.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:08.004 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:08.006+0000 7f7827a59a40 -1 Falling back to public interface 2026-02-21T01:41:08.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:08.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:08.654 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:08.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:08.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:08.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:08.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:08.949 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:08.950+0000 7f7827a59a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:41:09.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:09.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:09.761 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:41:09.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:41:09.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:09.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:09.968 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3402731761,v1:127.0.0.1:6801/3402731761] [v2:127.0.0.1:6802/3402731761,v1:127.0.0.1:6803/3402731761] exists,up 02c72d3a-8ffd-4deb-bce7-879dcb50aa41 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: test_wait_for_osd: run_osd td/ceph-helpers 1 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:09.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:09.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:09.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:09.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:09.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:41:09.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:41:09.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:09.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7eea5617-4300-47e8-a07d-3234122724e5 2026-02-21T01:41:09.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7eea5617-4300-47e8-a07d-3234122724e5' 2026-02-21T01:41:09.973 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 7eea5617-4300-47e8-a07d-3234122724e5 2026-02-21T01:41:09.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:09.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1DZlpqE0SOxAAJ310dyeFJGbUp0HOEBpCrg== 2026-02-21T01:41:09.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1DZlpqE0SOxAAJ310dyeFJGbUp0HOEBpCrg=="}' 2026-02-21T01:41:09.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7eea5617-4300-47e8-a07d-3234122724e5 -i td/ceph-helpers/1/new.json 2026-02-21T01:41:10.199 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:10.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:41:10.209 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1DZlpqE0SOxAAJ310dyeFJGbUp0HOEBpCrg== --osd-uuid 7eea5617-4300-47e8-a07d-3234122724e5 2026-02-21T01:41:10.225 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:10.226+0000 7f16d94eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:10.226 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:10.230+0000 7f16d94eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:10.227 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:10.230+0000 7f16d94eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:10.228 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:10.230+0000 7f16d94eba40 -1 bdev(0x55d419e94c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:10.228 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:10.230+0000 7f16d94eba40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:41:10.897 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:41:10.897 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:10.898 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:41:10.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:41:10.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:11.179 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:41:11.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:41:11.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:11.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:11.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:11.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:11.203 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:11.206+0000 7f2692f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:11.204 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:11.206+0000 7f2692f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:11.206 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:11.206+0000 7f2692f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:11.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:41:11.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:11.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:41:11.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:11.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:11.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:11.396 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:11.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:11.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:11.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:11.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:11.640 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:11.642+0000 7f2692f70a40 -1 Falling back to public interface 2026-02-21T01:41:12.296 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:12.298+0000 7f2692f70a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:41:12.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:12.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:12.609 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:12.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:12.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:12.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:12.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:13.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:13.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:13.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:41:13.831 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:41:13.832 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:13.832 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:14.044 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/615202952,v1:127.0.0.1:6811/615202952] [v2:127.0.0.1:6812/615202952,v1:127.0.0.1:6813/615202952] exists,up 7eea5617-4300-47e8-a07d-3234122724e5 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:992: test_wait_for_osd: wait_for_osd up 0 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:14.045 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:14.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:14.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:14.046 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:14.258 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3402731761,v1:127.0.0.1:6801/3402731761] [v2:127.0.0.1:6802/3402731761,v1:127.0.0.1:6803/3402731761] exists,up 02c72d3a-8ffd-4deb-bce7-879dcb50aa41 2026-02-21T01:41:14.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:14.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:14.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:14.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:993: test_wait_for_osd: wait_for_osd up 1 2026-02-21T01:41:14.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:14.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:41:14.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:14.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:14.260 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:14.260 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:14.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:14.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:14.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:14.472 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/615202952,v1:127.0.0.1:6811/615202952] [v2:127.0.0.1:6812/615202952,v1:127.0.0.1:6813/615202952] exists,up 7eea5617-4300-47e8-a07d-3234122724e5 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:994: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:14.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:14.781 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: wait_for_osd down 0 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:14.782 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T01:41:14.983 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 11 last_clean_interval [0,0) [v2:127.0.0.1:6800/3402731761,v1:127.0.0.1:6801/3402731761] [v2:127.0.0.1:6802/3402731761,v1:127.0.0.1:6803/3402731761] exists 02c72d3a-8ffd-4deb-bce7-879dcb50aa41 2026-02-21T01:41:14.983 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:14.983 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: TIMEOUT=1 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: wait_for_osd up 0 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 1 )) 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:14.984 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:15.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:16.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:16.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 1 )) 2026-02-21T01:41:16.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 1 2026-02-21T01:41:16.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: teardown td/ceph-helpers 2026-02-21T01:41:16.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:16.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:16.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:16.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:16.199 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:16.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:16.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:16.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:16.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:16.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:16.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:16.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:16.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:16.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:16.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:16.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:16.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:16.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:16.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:16.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:16.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:16.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:16.321 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:16.321 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.321 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.322 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_for_peered td/ceph-helpers 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1766: test_wait_for_peered: local dir=td/ceph-helpers 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1768: test_wait_for_peered: setup td/ceph-helpers 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:16.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:16.324 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:16.324 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:16.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:16.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:16.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:16.326 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:16.327 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:16.327 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:16.328 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:16.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:16.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:16.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:16.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:16.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:16.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:16.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:16.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:16.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:16.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:16.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:16.334 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.334 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:16.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:16.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:16.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:41:16.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:41:16.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.336 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:41:16.338 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1769: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:41:16.339 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:16.362 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-21T01:41:16.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:16.393 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:16.394 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:16.394 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.394 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.394 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:16.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:16.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:41:16.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:16.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:16.453 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:16.453 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.453 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.453 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:16.454 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:16.454 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1770: test_wait_for_peered: run_osd td/ceph-helpers 0 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:16.509 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:41:16.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:41:16.512 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:16.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ea6a9fc3-550c-4877-87f2-644714e119bc 2026-02-21T01:41:16.513 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ea6a9fc3-550c-4877-87f2-644714e119bc' 2026-02-21T01:41:16.513 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 ea6a9fc3-550c-4877-87f2-644714e119bc 2026-02-21T01:41:16.513 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:16.525 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8DZlp7AKrHxAALeVxoLa4ICsZyQ6zjt7tFQ== 2026-02-21T01:41:16.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8DZlp7AKrHxAALeVxoLa4ICsZyQ6zjt7tFQ=="}' 2026-02-21T01:41:16.526 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ea6a9fc3-550c-4877-87f2-644714e119bc -i td/ceph-helpers/0/new.json 2026-02-21T01:41:16.645 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:16.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:41:16.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8DZlp7AKrHxAALeVxoLa4ICsZyQ6zjt7tFQ== --osd-uuid ea6a9fc3-550c-4877-87f2-644714e119bc 2026-02-21T01:41:16.681 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:16.682+0000 7fe0a7fdba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:16.682 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:16.682+0000 7fe0a7fdba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:16.683 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:16.686+0000 7fe0a7fdba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:16.684 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:16.686+0000 7fe0a7fdba40 -1 bdev(0x559bcf32ec00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:16.684 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:16.686+0000 7fe0a7fdba40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:41:17.322 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:41:17.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:17.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:41:17.323 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:41:17.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:17.449 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:41:17.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:41:17.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:17.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:17.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:17.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:17.474 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:17.474+0000 7f704d5aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:17.476 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:17.478+0000 7f704d5aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:17.476 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:17.478+0000 7f704d5aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:17.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:17.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:17.897 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:17.898+0000 7f704d5aba40 -1 Falling back to public interface 2026-02-21T01:41:18.510 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:18.510+0000 7f704d5aba40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:41:18.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:18.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:18.677 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:18.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:18.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:18.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:18.785 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:19.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:19.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:19.787 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:41:19.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:41:19.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:19.788 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/225399228,v1:127.0.0.1:6801/225399228] [v2:127.0.0.1:6802/225399228,v1:127.0.0.1:6803/225399228] exists,up ea6a9fc3-550c-4877-87f2-644714e119bc 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1771: test_wait_for_peered: run_mgr td/ceph-helpers x 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:41:19.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:20.015 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:20.016 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:20.016 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:20.035 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1772: test_wait_for_peered: create_rbd_pool 2026-02-21T01:41:20.035 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:41:20.136 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:41:20.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:41:20.144 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:41:20.308 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:41:20.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:41:21.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1773: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1773: test_wait_for_peered: wait_for_clean 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 1 .1 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:41:21.594 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:41:21.595 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:41:21.595 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:41:21.595 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:41:21.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2026-02-21T01:41:21.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:41:21.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:41:21.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:41:21.623 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:41:21.623 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:41:21.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:41:21.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:41:21.738 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:41:21.739 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:41:21.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=0 2026-02-21T01:41:21.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 0 2026-02-21T01:41:21.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-0' 2026-02-21T01:41:21.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:41:21.819 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-0 2026-02-21T01:41:21.819 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:41:21.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:41:21.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-0 2026-02-21T01:41:21.821 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:41:21.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=0 2026-02-21T01:41:21.822 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 0 2026-02-21T01:41:21.823 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 0' 2026-02-21T01:41:21.824 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:21.932 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 0 -lt 0 2026-02-21T01:41:21.933 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:41:21.933 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:21.933 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:22.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 0 == 0 2026-02-21T01:41:22.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: sleep 1 2026-02-21T01:41:23.047 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:41:23.048 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:23.048 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:23.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:41:23.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:23.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:23.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:23.332 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:23.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:23.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:23.333 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:23.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:41:23.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:23.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:23.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:23.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:41:23.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-21T01:41:23.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-21T01:41:23.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-21T01:41:23.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:41:23.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-21T01:41:23.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:41:23.931 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:23.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:23.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:23.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:23.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:23.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:23.932 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:24.148 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:41:24.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:24.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:24.148 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:41:24.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:41:24.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:41:24.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:41:24.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 4 )) 2026-02-21T01:41:24.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:41:24.710 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-21T01:41:24.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:41:24.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:24.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:24.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:24.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:24.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:24.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:24.914 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:25.127 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:41:25.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:25.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:25.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:41:25.411 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:41:25.412 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:41:25.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:41:25.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:41:25.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 4 )) 2026-02-21T01:41:25.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:41:25.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-21T01:41:26.105 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:41:26.105 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:26.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:26.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:26.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:26.105 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:26.106 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:26.106 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:26.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:41:26.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:26.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:26.316 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:26.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:41:26.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:41:26.589 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:41:26.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:41:26.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:41:26.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:41:26.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:41:26.590 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:41:26.590 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:41:26.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:41:26.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:41:26.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 4 )) 2026-02-21T01:41:26.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-21T01:41:26.875 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.3 2026-02-21T01:41:27.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-21T01:41:27.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:27.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:27.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:27.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:27.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:27.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:27.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:27.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-21T01:41:27.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:27.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:27.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:27.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 4 2026-02-21T01:41:27.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-21T01:41:27.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-21T01:41:27.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T01:41:27.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T01:41:27.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T01:41:27.685 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T01:41:27.686 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T01:41:27.686 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T01:41:27.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T01:41:27.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T01:41:27.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 4 >= 4 )) 2026-02-21T01:41:27.974 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: ceph report 2026-02-21T01:41:28.242 INFO:tasks.workunit.client.0.trial106.stdout:{ 2026-02-21T01:41:28.242 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_fingerprint": "96ba0cd8-0ab2-4df7-9432-b699961f957f", 2026-02-21T01:41:28.242 INFO:tasks.workunit.client.0.trial106.stdout: "version": "20.2.0-681-g36d388f2", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "commit": "36d388f27052dd940bdbaf543bb05bdba0acc155", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "timestamp": "2026-02-21T01:41:28.246759+0000", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "tag": "", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "health": { 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "status": "HEALTH_WARN", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "checks": { 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "PG_DEGRADED": { 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "count": 2 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [ 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.243 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "TOO_FEW_OSDS": { 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "severity": "HEALTH_WARN", 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "summary": { 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "count": 2 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "detail": [], 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "muted": false 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "mutes": [] 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "monmap_first_committed": 1, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "monmap_last_committed": 1, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "monmap": { 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "fsid": "640ad1de-49d5-4d28-b121-3f053a140605", 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:41:16.360511Z", 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "created": "2026-02-21T01:41:16.360511Z", 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "min_mon_release": 20, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "min_mon_release_name": "tentacle", 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "election_strategy": 1, 2026-02-21T01:41:28.244 INFO:tasks.workunit.client.0.trial106.stdout: "disallowed_leaders": "", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode": false, 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "tiebreaker_mon": "", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "removed_ranks": "", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "features": { 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "persistent": [ 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "kraken", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "luminous", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "mimic", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap-prune", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "nautilus", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "octopus", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "pacific", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "elector-pinging", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "quincy", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "reef", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "squid", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "tentacle", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "nvmeof_beacon_diff" 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "optional": [] 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "mons": [ 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "rank": 0, 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "name": "a", 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "public_addrs": { 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:41:28.245 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:7109", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 0 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:7109/0", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "public_addr": "127.0.0.1:7109/0", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "priority": 0, 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 0, 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "crush_location": "{}" 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "quorum": [ 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: 0 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap": { 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 12, 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "fsid": "640ad1de-49d5-4d28-b121-3f053a140605", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "created": "2026-02-21T01:41:16.446214+0000", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:41:21.594270+0000", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "last_up_change": "2026-02-21T01:41:19.135204+0000", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "last_in_change": "2026-02-21T01:41:16.629186+0000", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "flags_num": 5799936, 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "flags_set": [ 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "pglog_hardlimit", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snapdirs", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "recovery_deletes", 2026-02-21T01:41:28.246 INFO:tasks.workunit.client.0.trial106.stdout: "sortbitwise" 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "crush_version": 3, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "full_ratio": 0.99000000953674316, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "backfillfull_ratio": 0.99000000953674316, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "nearfull_ratio": 0.85000002384185791, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_snapshot": "", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pool_max": 1, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "max_osd": 1, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "require_min_compat_client": "luminous", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "min_compat_client": "jewel", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "require_osd_release": "tentacle", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "allow_crimson": false, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pools": [ 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pool_name": "rbd", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "create_time": "2026-02-21T01:41:20.251984+0000", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "flags": 8193, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "type": 1, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "size": 2, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "min_size": 1, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "crush_rule": 0, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_count": 0, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_target": 0, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_barrier": 0, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "is_stretch_pool": false, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "object_hash": 2, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pg_autoscale_mode": "off", 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num": 4, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pg_placement_num": 4, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pg_placement_num_target": 4, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num_target": 4, 2026-02-21T01:41:28.247 INFO:tasks.workunit.client.0.trial106.stdout: "pg_num_pending": 4, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_pg_merge_meta": { 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "source_pgid": "0.0", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "ready_epoch": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_started": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_clean": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "source_version": "0'0", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "target_version": "0'0" 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_change": "12", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend": "0", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend_prenautilus": "0", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "last_force_op_resend_preluminous": "0", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "auid": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "snap_mode": "selfmanaged", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "snap_seq": 2, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "snap_epoch": 12, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "pool_snaps": [], 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "removed_snaps": "[]", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "quota_max_bytes": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "quota_max_objects": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "tiers": [], 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "tier_of": -1, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "read_tier": -1, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "write_tier": -1, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "cache_mode": "none", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "target_max_bytes": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "target_max_objects": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_dirty_ratio_micro": 400000, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "cache_target_full_ratio_micro": 800000, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "cache_min_flush_age": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "cache_min_evict_age": 0, 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "erasure_code_profile": "", 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_params": { 2026-02-21T01:41:28.248 INFO:tasks.workunit.client.0.trial106.stdout: "type": "none" 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_period": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_count": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "use_gmt_hitset": true, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "min_read_recency_for_promote": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "min_write_recency_for_promote": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_grade_decay_rate": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "hit_set_search_last_n": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "grade_table": [], 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "stripe_width": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "expected_num_objects": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "fast_read": false, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "nonprimary_shards": "{}", 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "options": {}, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "application_metadata": { 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "rbd": {} 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "read_balance": { 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "score_type": "Fair distribution", 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "score_acting": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "score_stable": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "optimal_score": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "raw_score_acting": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "raw_score_stable": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "primary_affinity_weighted": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "average_primary_affinity": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "average_primary_affinity_weighted": 1 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "osds": [ 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "uuid": "ea6a9fc3-550c-4877-87f2-644714e119bc", 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "up": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "in": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "primary_affinity": 1, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "last_clean_begin": 0, 2026-02-21T01:41:28.249 INFO:tasks.workunit.client.0.trial106.stdout: "last_clean_end": 0, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 5, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "up_thru": 7, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "down_at": 0, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "lost_at": 0, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "public_addrs": { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6800", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6801", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_addrs": { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6802", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6803", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_back_addrs": { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6806", 2026-02-21T01:41:28.250 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6807", 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_front_addrs": { 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "addrvec": [ 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v2", 2026-02-21T01:41:28.251 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6804", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "type": "v1", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "addr": "127.0.0.1:6805", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "nonce": 225399228 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "public_addr": "127.0.0.1:6801/225399228", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "cluster_addr": "127.0.0.1:6803/225399228", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_back_addr": "127.0.0.1:6807/225399228", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "heartbeat_front_addr": "127.0.0.1:6805/225399228", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "state": [ 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "exists", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "up" 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "osd_xinfo": [ 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "down_stamp": "0.000000", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "laggy_probability": 0, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "laggy_interval": 0, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "features": 4544132024016699391, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "old_weight": 0, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "last_purged_snaps_scrub": "2026-02-21T01:41:19.553770+0000", 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "dead_epoch": 0 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap": [], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap_items": [], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "pg_upmap_primaries": [], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "pg_temp": [], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "primary_temp": [], 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "blocklist": {}, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "range_blocklist": {}, 2026-02-21T01:41:28.252 INFO:tasks.workunit.client.0.trial106.stdout: "erasure_code_profiles": { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "default": { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "k": "2", 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "m": "2", 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "plugin": "isa", 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "technique": "reed_sol_van" 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "removed_snaps_queue": [ 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "snaps": [ 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "begin": 2, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "length": 1 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "new_removed_snaps": [ 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "snaps": [ 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "begin": 2, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "length": 1 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "new_purged_snaps": [], 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "crush_node_flags": {}, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "device_class_flags": {}, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode": { 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode_enabled": false, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_bucket_count": 0, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "degraded_stretch_mode": 0, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "recovering_stretch_mode": 0, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "stretch_mode_bucket": 0 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.253 INFO:tasks.workunit.client.0.trial106.stdout: "osd_metadata": [ 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "arch": "x86_64", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "back_addr": "[v2:127.0.0.1:6802/225399228,v1:127.0.0.1:6803/225399228]", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "back_iface": "", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs": "1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_dedicated_db": "0", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_dedicated_wal": "0", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluefs_single_shared_device": "1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_allocation_from_file": "1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_allocator": "hybrid", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_access_mode": "file", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_block_size": "4096", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_devices": "nvme0n1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_driver": "KernelDevice", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_optimal_io_size": "4096", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_rotational": "0", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_size": "107374182400", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_support_discard": "1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_bdev_type": "ssd", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_min_alloc_size": "4096", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_onode_segmentation": "inactive", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "bluestore_write_mode": "classic", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_release": "tentacle", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version": "ceph version 20.2.0-681-g36d388f2 (36d388f27052dd940bdbaf543bb05bdba0acc155) tentacle (stable - None)", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version_short": "20.2.0-681-g36d388f2", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "ceph_version_when_created": "ceph version 20.2.0-681-g36d388f2 (36d388f27052dd940bdbaf543bb05bdba0acc155) tentacle (stable - None)", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "cpu": "AMD EPYC 4464P 12-Core Processor", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "created_at": "2026-02-21T01:41:17.224183Z", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "default_device_class": "ssd", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "device_ids": "nvme0n1=Micron_7450_MTFDKBA960TFR_25174FCD6E72", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "device_paths": "nvme0n1=/dev/disk/by-path/pci-0000:02:00.0-nvme-1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "devices": "nvme0n1", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "distro": "ubuntu", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "distro_description": "Ubuntu 22.04.5 LTS", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "distro_version": "22.04", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "front_addr": "[v2:127.0.0.1:6800/225399228,v1:127.0.0.1:6801/225399228]", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "front_iface": "", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/225399228,v1:127.0.0.1:6807/225399228]", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/225399228,v1:127.0.0.1:6805/225399228]", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "hostname": "trial106", 2026-02-21T01:41:28.254 INFO:tasks.workunit.client.0.trial106.stdout: "journal_rotational": "0", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "kernel_description": "#180-Ubuntu SMP Fri Jan 9 16:10:31 UTC 2026", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "kernel_version": "5.15.0-170-generic", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "mem_swap_kb": "8388604", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "mem_total_kb": "130961696", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "objectstore_numa_node": "0", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "objectstore_numa_nodes": "0", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "os": "Linux", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osd_data": "td/ceph-helpers/0", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osd_objectstore": "bluestore", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osdspec_affinity": "", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "rotational": "0" 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_clean_epochs": { 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "min_last_epoch_clean": 0, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "last_epoch_clean": { 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "per_pool": [] 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osd_epochs": [ 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 5 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_first_committed": 1, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "osdmap_last_committed": 12, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "crushmap": { 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "devices": [ 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "name": "osd.0", 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "class": "ssd" 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "types": [ 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 0, 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: "name": "osd" 2026-02-21T01:41:28.255 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "host" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 2, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "chassis" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 3, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "rack" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 4, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "row" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 5, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "pdu" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 6, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "pod" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 7, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "room" 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 8, 2026-02-21T01:41:28.256 INFO:tasks.workunit.client.0.trial106.stdout: "name": "datacenter" 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 9, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "name": "zone" 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 10, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "name": "region" 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "name": "root" 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "buckets": [ 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "id": -1, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "name": "default", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "root", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "id": -3, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "id": -2, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "name": "default~ssd", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 11, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "root", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:41:28.257 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "id": -4, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "id": -3, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "name": "trial106", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "host", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "id": -4, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "name": "trial106~ssd", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "type_id": 1, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "type_name": "host", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "alg": "straw2", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "hash": "rjenkins1", 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "items": [ 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "id": 0, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "weight": 6402, 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "pos": 0 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.258 INFO:tasks.workunit.client.0.trial106.stdout: "rules": [ 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "rule_id": 0, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "rule_name": "replicated_rule", 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "type": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "steps": [ 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "op": "take", 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "item": -1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "item_name": "default" 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "op": "choose_firstn", 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "num": 0, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "type": "osd" 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "op": "emit" 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: ] 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "tunables": { 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "choose_local_tries": 0, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "choose_local_fallback_tries": 0, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "choose_total_tries": 50, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_descend_once": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_vary_r": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "chooseleaf_stable": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "msr_descents": 100, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "msr_collision_tries": 100, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "straw_calc_version": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "allowed_bucket_algs": 54, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "profile": "jewel", 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "optimal_tunables": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "legacy_tunables": 0, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "minimum_required_version": "jewel", 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables2": 1, 2026-02-21T01:41:28.259 INFO:tasks.workunit.client.0.trial106.stdout: "has_v2_rules": 0, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables3": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "has_v3_rules": 0, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "has_v4_buckets": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "require_feature_tunables5": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "has_v5_rules": 0, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "has_msr_rules": 0 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "choose_args": {} 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "fsmap": { 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "btime": "2026-02-21T01:41:16:446011+0000", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "default_fscid": -1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "compat": { 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "compat": {}, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "ro_compat": {}, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "incompat": { 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_1": "base v0.20", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_2": "client writeable ranges", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_3": "default file layouts on dirs", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_4": "dir inode in separate object", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_5": "mds uses versioned encoding", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_6": "dirfrag is stored in omap", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_8": "no anchor table", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_9": "file layout v2", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_10": "snaprealm v2", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_11": "minor log segments", 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_12": "quiesce subvolumes" 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "feature_flags": { 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "enable_multiple": true, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "ever_enabled_multiple": true 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "standbys": [], 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "filesystems": [] 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "mdsmap_first_committed": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "mdsmap_last_committed": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "auth": { 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "first_committed": 1, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "last_committed": 3, 2026-02-21T01:41:28.260 INFO:tasks.workunit.client.0.trial106.stdout: "num_secrets": 1 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg": 4, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_active": 4, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_unknown": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_osd": 1, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "pool_sum": { 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "stat_sum": { 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes": 19, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects": 2, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_clones": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_copies": 4, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing_on_primary": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_degraded": 2, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_misplaced": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_unfound": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_dirty": 2, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_whiteouts": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_read": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_read_kb": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_write": 2, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_write_kb": 2, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_scrub_errors": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_shallow_scrub_errors": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_deep_scrub_errors": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_recovered": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_recovered": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_keys_recovered": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_omap": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_hit_set_archive": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_hit_set_archive": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_kb": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_kb": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_promote": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_high": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_low": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_some": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_full": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_pinned": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_legacy_snapsets": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_large_omap_objects": 0, 2026-02-21T01:41:28.261 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_manifest": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_bytes": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_keys": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_repaired": 0 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "store_stats": { 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "total": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "available": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 0 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "log_size": 3, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "ondisk_log_size": 3, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "up": 4, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "acting": 4, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_store_stats": 0 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "osd_sum": { 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_pgs": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_osds": 1, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_osds": 1, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_omap_osds": 1, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "kb": 104857600, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used": 27048, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_data": 88, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_omap": 7, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_meta": 26936, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "kb_avail": 104830552, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "statfs": { 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "total": 107374182400, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "available": 107346485248, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 90112, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 18391, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:41:28.262 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 7483, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 27583173 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "hb_peers": [], 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "snap_trim_queue_len": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "num_snap_trimming": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "num_shards_repaired": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "op_queue_age_hist": { 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "histogram": [], 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "upper_bound": 1 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "perf_stat": { 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ms": 2, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ms": 2, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ns": 2000000, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ns": 2000000 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "alerts": [], 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "network_ping_times": [] 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "osd_sum_by_class": { 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "ssd": { 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "up_from": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "num_pgs": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "num_osds": 1, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_osds": 1, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "num_per_pool_omap_osds": 1, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "kb": 104857600, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used": 27048, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_data": 88, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_omap": 7, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "kb_used_meta": 26936, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "kb_avail": 104830552, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "statfs": { 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "total": 107374182400, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "available": 107346485248, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 90112, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 18391, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:41:28.263 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 7483, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 27583173 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "hb_peers": [], 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "snap_trim_queue_len": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_snap_trimming": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_shards_repaired": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "op_queue_age_hist": { 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "histogram": [], 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "upper_bound": 1 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "perf_stat": { 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ms": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ms": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "commit_latency_ns": 2000000, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "apply_latency_ns": 2000000 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "alerts": [], 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "network_ping_times": [] 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "pool_stats": [ 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "poolid": 1, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg": 4, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "stat_sum": { 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes": 19, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_clones": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_object_copies": 4, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing_on_primary": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_missing": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_degraded": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_misplaced": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_unfound": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_dirty": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_whiteouts": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_read": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_read_kb": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_write": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_write_kb": 2, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_scrub_errors": 0, 2026-02-21T01:41:28.264 INFO:tasks.workunit.client.0.trial106.stdout: "num_shallow_scrub_errors": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_deep_scrub_errors": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_recovered": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_recovered": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_keys_recovered": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_omap": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_hit_set_archive": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_bytes_hit_set_archive": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_kb": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_kb": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_promote": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_high": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_flush_mode_low": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_some": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_evict_mode_full": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_pinned": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_legacy_snapsets": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_large_omap_objects": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_manifest": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_bytes": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_omap_keys": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_objects_repaired": 0 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "store_stats": { 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "total": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "available": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "internally_reserved": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "allocated": 4096, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "data_stored": 19, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_allocated": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "data_compressed_original": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "omap_allocated": 0, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "internal_metadata": 0 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "log_size": 3, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "ondisk_log_size": 3, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "up": 4, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "acting": 4, 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "num_store_stats": 1 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.265 INFO:tasks.workunit.client.0.trial106.stdout: "osd_stats": [ 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "seq": 21474836481 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_by_state": [ 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "state": "active+undersized+degraded", 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num": 2 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "state": "active+undersized", 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num": 2 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "pool_pg_unavailable_map": [ 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "poolid": "1", 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "pgs": [] 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num_pg_by_osd": [ 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "osd": 0, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num_primary_pg": 4, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num_acting_pg": 4, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "num_up_not_acting_pg": 0 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snaps": [ 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "pool": 1, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "purged_snaps": {} 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: ], 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "servicemap": { 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "epoch": 1, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "modified": "2026-02-21T01:41:16.446398+0000", 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "services": {} 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: }, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "mgrstat_first_committed": 1, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "mgrstat_last_committed": 4, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "logm_first_committed": 1, 2026-02-21T01:41:28.266 INFO:tasks.workunit.client.0.trial106.stdout: "logm_last_committed": 56, 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout: "paxos": { 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout: "first_committed": 1, 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout: "last_committed": 81, 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout: "last_pn": 0, 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout: "accepted_pn": 0 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout: } 2026-02-21T01:41:28.267 INFO:tasks.workunit.client.0.trial106.stdout:} 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:report 2135189873 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: return 1 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1774: test_wait_for_peered: run_osd td/ceph-helpers 1 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:28.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ab15ba33-8f11-495f-a30d-0b1d387248d6 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ab15ba33-8f11-495f-a30d-0b1d387248d6' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 ab15ba33-8f11-495f-a30d-0b1d387248d6 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBIDZlpXqpSEBAAc/7QloC2ehIOEGsts49XTw== 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBIDZlpXqpSEBAAc/7QloC2ehIOEGsts49XTw=="}' 2026-02-21T01:41:28.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ab15ba33-8f11-495f-a30d-0b1d387248d6 -i td/ceph-helpers/1/new.json 2026-02-21T01:41:28.487 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:28.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T01:41:28.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIDZlpXqpSEBAAc/7QloC2ehIOEGsts49XTw== --osd-uuid ab15ba33-8f11-495f-a30d-0b1d387248d6 2026-02-21T01:41:28.513 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:28.514+0000 7fcc46faaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:28.515 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:28.518+0000 7fcc46faaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:28.516 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:28.518+0000 7fcc46faaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:28.516 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:28.518+0000 7fcc46faaa40 -1 bdev(0x556707430c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:28.517 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:28.518+0000 7fcc46faaa40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T01:41:29.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T01:41:29.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:29.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:41:29.139 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:41:29.139 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:29.419 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:41:29.420 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:41:29.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:29.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:29.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:29.420 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:29.443 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:29.445+0000 7f2f70c4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:29.444 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:29.445+0000 7f2f70c4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:29.446 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:29.445+0000 7f2f70c4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:29.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:41:29.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:29.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:41:29.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:29.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:29.636 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:29.636 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:29.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:29.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:29.637 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:29.813 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:29.813+0000 7f2f70c4ea40 -1 Falling back to public interface 2026-02-21T01:41:29.849 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:30.425 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:30.425+0000 7f2f70c4ea40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:41:30.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:30.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:30.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:30.851 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:30.851 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:30.851 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:41:31.069 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3059077548,v1:127.0.0.1:6811/3059077548] [v2:127.0.0.1:6812/3059077548,v1:127.0.0.1:6813/3059077548] exists,up ab15ba33-8f11-495f-a30d-0b1d387248d6 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1775: test_wait_for_peered: wait_for_peered 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1731: wait_for_peered: local cmd= 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1732: wait_for_peered: local num_peered=-1 2026-02-21T01:41:31.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1733: wait_for_peered: local cur_peered 2026-02-21T01:41:31.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: get_timeout_delays 90 .1 2026-02-21T01:41:31.072 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:41:31.072 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:41:31.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:41:31.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:41:31.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:41:31.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:41:31.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: local -a delays 2026-02-21T01:41:31.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1735: wait_for_peered: local -i loop=0 2026-02-21T01:41:31.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: wait_for_peered: flush_pg_stats 2026-02-21T01:41:31.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:41:31.171 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:41:31.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:41:31.393 INFO:tasks.workunit.client.0.trial106.stderr:1' 2026-02-21T01:41:31.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:41:31.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:41:31.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:41:31.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-21T01:41:31.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-21T01:41:31.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T01:41:31.472 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:41:31.472 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:41:31.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=0 2026-02-21T01:41:31.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 0 2026-02-21T01:41:31.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-0' 2026-02-21T01:41:31.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:41:31.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-21T01:41:31.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:41:31.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:41:31.554 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-21T01:41:31.555 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:41:31.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-21T01:41:31.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T01:41:31.556 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836484 2026-02-21T01:41:31.556 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:31.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-21T01:41:31.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:41:32.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:41:32.775 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:32.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-21T01:41:32.992 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:41:32.993 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-0 2026-02-21T01:41:32.993 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:41:32.994 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:41:32.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-0 2026-02-21T01:41:32.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:41:32.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=0 2026-02-21T01:41:32.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 0' 2026-02-21T01:41:32.997 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 0 2026-02-21T01:41:32.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:41:33.208 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 68719476737 -lt 0 2026-02-21T01:41:33.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: get_num_pgs 2026-02-21T01:41:33.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:33.209 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:33.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: test 4 == 0 2026-02-21T01:41:33.491 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: wait_for_peered: true 2026-02-21T01:41:33.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: get_num_active_or_peered 2026-02-21T01:41:33.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-21T01:41:33.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-21T01:41:33.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-21T01:41:33.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-21T01:41:33.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:33.701 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: cur_peered=4 2026-02-21T01:41:33.701 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: get_num_pgs 2026-02-21T01:41:33.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:33.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: test 4 = 4 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: break 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_peered: return 0 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1776: test_wait_for_peered: teardown td/ceph-helpers 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:33.979 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:33.980 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:33.980 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:33.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:33.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:33.980 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:34.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:34.090 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:34.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:34.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:34.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:34.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:34.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:34.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:34.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:34.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:34.095 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:34.096 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:34.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:34.097 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:34.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:34.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:34.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_for_scrub td/ceph-helpers 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: test_wait_for_scrub: local dir=td/ceph-helpers 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: test_wait_for_scrub: setup td/ceph-helpers 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:34.107 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:34.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:34.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:34.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:34.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:34.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:34.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:34.109 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:34.110 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:34.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:34.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:34.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:34.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:34.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:34.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:34.113 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:34.114 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:34.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:34.116 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:34.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:34.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:34.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:34.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:34.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:41:34.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:41:34.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.120 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:41:34.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:41:34.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:41:34.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:41:34.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:34.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:41:34.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:34.145 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:34.145 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:34.146 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.146 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.146 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:34.146 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:34.174 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:41:34.175 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:41:34.175 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:34.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:34.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:41:34.176 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:34.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:41:34.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:34.235 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:34.236 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:34.237 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.237 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.237 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:34.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:34.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:41:34.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: test_wait_for_scrub: run_mgr td/ceph-helpers x 2026-02-21T01:41:34.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:41:34.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:41:34.292 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:41:34.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:41:34.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:41:34.293 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:41:34.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:41:34.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:34.407 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:34.408 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:34.408 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.408 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.408 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:34.408 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:34.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: test_wait_for_scrub: run_osd td/ceph-helpers 0 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:34.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:41:34.428 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:41:34.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:34.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8335426e-323f-4224-b03f-88859ae65ecc 2026-02-21T01:41:34.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8335426e-323f-4224-b03f-88859ae65ecc' 2026-02-21T01:41:34.430 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 8335426e-323f-4224-b03f-88859ae65ecc 2026-02-21T01:41:34.430 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:34.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBODZlpQleVGhAA/SeFRpERDcrl/yqHDYodLw== 2026-02-21T01:41:34.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBODZlpQleVGhAA/SeFRpERDcrl/yqHDYodLw=="}' 2026-02-21T01:41:34.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8335426e-323f-4224-b03f-88859ae65ecc -i td/ceph-helpers/0/new.json 2026-02-21T01:41:34.549 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:34.557 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:41:34.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBODZlpQleVGhAA/SeFRpERDcrl/yqHDYodLw== --osd-uuid 8335426e-323f-4224-b03f-88859ae65ecc 2026-02-21T01:41:34.574 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:34.573+0000 7fdffed6ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:34.575 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:34.577+0000 7fdffed6ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:34.576 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:34.577+0000 7fdffed6ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:34.576 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:34.577+0000 7fdffed6ea40 -1 bdev(0x56035c6c4c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:34.576 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:34.577+0000 7fdffed6ea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:41:35.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:41:35.263 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:35.263 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:41:35.263 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:41:35.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:35.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:41:35.374 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:41:35.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:35.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:35.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:35.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:35.389 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:35.389+0000 7f071d6e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:35.390 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:35.389+0000 7f071d6e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:35.391 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:35.393+0000 7f071d6e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:35.488 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:35.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:35.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:35.597 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:35.796 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:35.797+0000 7f071d6e2a40 -1 Falling back to public interface 2026-02-21T01:41:36.434 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:36.433+0000 7f071d6e2a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:41:36.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:36.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:36.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:36.599 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:36.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:36.599 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:36.730 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:37.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:37.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:37.732 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:41:37.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:41:37.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:37.733 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:37.955 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/714461732,v1:127.0.0.1:6801/714461732] [v2:127.0.0.1:6802/714461732,v1:127.0.0.1:6803/714461732] exists,up 8335426e-323f-4224-b03f-88859ae65ecc 2026-02-21T01:41:37.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:37.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:37.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:37.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: test_wait_for_scrub: create_rbd_pool 2026-02-21T01:41:37.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:41:38.163 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:41:38.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:41:38.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:41:38.416 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:41:38.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:41:39.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:41:39.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: test_wait_for_scrub: wait_for_clean 2026-02-21T01:41:39.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:41:39.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:41:39.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:41:39.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:41:39.705 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:41:39.705 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:41:39.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:41:39.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:41:39.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:41:39.798 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:41:39.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:41:39.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:41:39.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:41:39.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:41:39.799 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:41:40.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:41:40.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:41:40.011 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:41:40.011 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:41:40.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:41:40.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:41:40.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:41:40.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:41:40.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:41:40.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:41:40.092 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:41:40.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:41:40.093 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:41:40.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:41:40.095 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:41:40.095 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:41:40.095 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:40.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:41:40.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:41:41.313 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:41:41.313 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:41.519 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:41:41.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:41:41.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:41.520 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:41.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:41:41.790 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:41.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:41.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:41.790 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:41.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:41.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:41.791 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:41.996 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:41:41.996 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:41.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:41.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:42.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:41:42.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:41:42.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:41:42.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: test_wait_for_scrub: local pgid=1.0 2026-02-21T01:41:42.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: test_wait_for_scrub: ceph pg repair 1.0 2026-02-21T01:41:42.468 INFO:tasks.workunit.client.0.trial106.stderr:instructing pg 1.0 on osd.0 to repair 2026-02-21T01:41:42.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: test_wait_for_scrub: get_last_scrub_stamp 1.0 2026-02-21T01:41:42.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:42.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:42.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:42.478 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:42.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: test_wait_for_scrub: local last_scrub=2026-02-21T01:41:38.368134+0000 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2088: test_wait_for_scrub: wait_for_scrub 1.0 2026-02-21T01:41:38.368134+0000 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-21T01:41:38.368134+0000 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:42.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:42.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:41:38.368134+0000 '>' 2026-02-21T01:41:38.368134+0000 2026-02-21T01:41:42.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:41:43.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:41:43.897 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:41:43.897 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:41:43.897 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:43.897 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:43.898 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:43.898 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:44.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:41:38.368134+0000 '>' 2026-02-21T01:41:38.368134+0000 2026-02-21T01:41:44.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:41:45.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:41:45.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-21T01:41:45.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:41:45.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:45.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:45.108 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:45.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:45.309 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:41:42.463406+0000 '>' 2026-02-21T01:41:38.368134+0000 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2089: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:45.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:45.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:45.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: test_wait_for_scrub: get_last_scrub_stamp 1.0 2026-02-21T01:41:45.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:45.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:45.418 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:45.419 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:45.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2090: test_wait_for_scrub: last_scrub=2026-02-21T01:41:42.463406+0000 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2091: test_wait_for_scrub: TIMEOUT=2 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2091: test_wait_for_scrub: wait_for_scrub 1.0 2026-02-21T01:41:42.463406+0000 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-21T01:41:42.463406+0000 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 2 )) 2026-02-21T01:41:45.633 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:41:45.634 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:45.634 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:45.634 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:45.634 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:45.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:41:42.463406+0000 '>' 2026-02-21T01:41:42.463406+0000 2026-02-21T01:41:45.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:41:46.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:41:46.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 2 )) 2026-02-21T01:41:46.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T01:41:46.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T01:41:46.843 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T01:41:46.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T01:41:46.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T01:41:47.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-21T01:41:42.463406+0000 '>' 2026-02-21T01:41:42.463406+0000 2026-02-21T01:41:47.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 2 )) 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: wait_for_scrub: return 1 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: test_wait_for_scrub: teardown td/ceph-helpers 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:48.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:48.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:48.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:48.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:48.055 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:48.165 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:48.165 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:48.166 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:48.166 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:48.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:48.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:48.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:48.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:48.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:48.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:48.169 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:48.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:48.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:48.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:48.177 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:48.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.178 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: for func in $funcs 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1492: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T01:41:48.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:48.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:48.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:48.183 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:48.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:48.184 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:48.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:48.185 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:48.185 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:48.186 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:48.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:48.187 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:48.187 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:48.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:48.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:48.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:48.190 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:48.190 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.191 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:48.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:48.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:48.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T01:41:48.193 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:41:48.193 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.193 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.193 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.61441 2026-02-21T01:41:48.195 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:41:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:41:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:41:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T01:41:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1495: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T01:41:48.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:41:48.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:41:48.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:41:48.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T01:41:48.197 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:48.218 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:41:48.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:48.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:48.219 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:48.219 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.219 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:48.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T01:41:48.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:41:48.248 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:41:48.248 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:48.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:48.249 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:41:48.250 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:41:48.250 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:48.250 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:48.251 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get fsid 2026-02-21T01:41:48.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:41:48.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:48.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:48.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:41:48.305 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:41:48.305 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:48.306 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:48.306 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:48.307 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:48.307 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.307 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.307 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.61441/ceph-mon.a.asok 2026-02-21T01:41:48.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:48.307 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.61441/ceph-mon.a.asok config get mon_host 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1496: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T01:41:48.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:41:48.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:41:48.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:48.471 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:48.471 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:48.472 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.472 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.472 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:48.473 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:48.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:48.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-21T01:41:48.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T01:41:48.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:48.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:41:48.494 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:48.495 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T01:41:48.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:41:48.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T01:41:48.498 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:48.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=95688198-b478-4a62-9355-8d1239b88580 2026-02-21T01:41:48.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 95688198-b478-4a62-9355-8d1239b88580' 2026-02-21T01:41:48.498 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 95688198-b478-4a62-9355-8d1239b88580 2026-02-21T01:41:48.499 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:48.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBcDZlpza6VHhAAuTMbY56ShiQJrz3XYP9/xQ== 2026-02-21T01:41:48.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBcDZlpza6VHhAAuTMbY56ShiQJrz3XYP9/xQ=="}' 2026-02-21T01:41:48.508 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 95688198-b478-4a62-9355-8d1239b88580 -i td/ceph-helpers/0/new.json 2026-02-21T01:41:48.624 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:48.632 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T01:41:48.633 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcDZlpza6VHhAAuTMbY56ShiQJrz3XYP9/xQ== --osd-uuid 95688198-b478-4a62-9355-8d1239b88580 2026-02-21T01:41:48.650 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:48.652+0000 7fd17b61fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:48.651 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:48.656+0000 7fd17b61fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:48.652 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:48.656+0000 7fd17b61fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:48.652 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:48.656+0000 7fd17b61fa40 -1 bdev(0x555c58cbcc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:48.652 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:48.656+0000 7fd17b61fa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T01:41:49.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T01:41:49.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:49.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:41:49.434 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:41:49.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:49.553 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:41:49.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:41:49.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=640ad1de-49d5-4d28-b121-3f053a140605 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.61441/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:41:49.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:49.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:49.554 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:49.578 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:49.580+0000 7fa5dc8cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:49.580 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:49.584+0000 7fa5dc8cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:49.581 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:49.584+0000 7fa5dc8cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:49.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:41:49.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:49.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:49.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:49.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:49.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:49.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:49.661 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:49.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:49.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:49.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:49.985 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:49.988+0000 7fa5dc8cea40 -1 Falling back to public interface 2026-02-21T01:41:50.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:50.768 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:50.768 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:50.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:50.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:51.039 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:51.040+0000 7fa5dc8cea40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:41:51.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:51.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:51.880 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:41:51.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:41:51.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:51.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:52.084 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1292115875,v1:127.0.0.1:6801/1292115875] [v2:127.0.0.1:6802/1292115875,v1:127.0.0.1:6803/1292115875] exists,up 95688198-b478-4a62-9355-8d1239b88580 2026-02-21T01:41:52.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:41:52.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:41:52.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:41:52.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1498: test_wait_osd_id_used_by_pgs: create_rbd_pool 2026-02-21T01:41:52.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T01:41:52.281 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' does not exist 2026-02-21T01:41:52.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T01:41:52.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T01:41:52.559 INFO:tasks.workunit.client.0.trial106.stderr:pool 'rbd' already exists 2026-02-21T01:41:52.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:41:53.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T01:41:53.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1499: test_wait_osd_id_used_by_pgs: wait_for_clean 2026-02-21T01:41:53.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:41:53.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:41:53.855 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:41:53.855 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:41:53.856 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:41:53.856 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:41:53.856 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:41:53.856 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:41:53.856 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:41:53.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:41:53.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:41:53.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:41:53.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:41:53.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:41:53.952 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:41:54.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-21T01:41:54.167 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:41:54.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:41:54.168 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:41:54.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-21T01:41:54.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-21T01:41:54.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T01:41:54.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:41:54.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-21T01:41:54.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:41:54.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:41:54.246 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-21T01:41:54.246 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:41:54.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-21T01:41:54.248 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836483 2026-02-21T01:41:54.248 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T01:41:54.248 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:54.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T01:41:54.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:41:55.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:41:55.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:41:55.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T01:41:55.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:41:55.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:55.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:55.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:41:55.955 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:41:56.161 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T01:41:56.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:41:56.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:41:56.162 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1500: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1475: wait_osd_id_used_by_pgs: local id=0 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1476: wait_osd_id_used_by_pgs: local count=8 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1478: wait_osd_id_used_by_pgs: status=1 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1479: wait_osd_id_used_by_pgs: (( i=0 )) 2026-02-21T01:41:56.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1479: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1480: wait_osd_id_used_by_pgs: echo 0 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1481: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1481: wait_osd_id_used_by_pgs: grep -c 0 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-21T01:41:56.436 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2026-02-21T01:41:56.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1481: wait_osd_id_used_by_pgs: test 8 = 8 2026-02-21T01:41:56.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: status=0 2026-02-21T01:41:56.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: break 2026-02-21T01:41:56.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: return 0 2026-02-21T01:41:56.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1475: wait_osd_id_used_by_pgs: local id=123 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1476: wait_osd_id_used_by_pgs: local count=5 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1478: wait_osd_id_used_by_pgs: status=1 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1479: wait_osd_id_used_by_pgs: (( i=0 )) 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1479: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: return 1 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1502: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:56.640 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:56.641 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:56.747 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:56.748 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:56.748 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:56.749 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:56.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:56.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:56.751 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:56.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:56.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:56.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:56.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:56.754 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:56.755 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:56.755 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T01:41:56.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:56.759 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.760 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.61441 2026-02-21T01:41:56.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.61441 2026-02-21T01:41:56.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:56.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:56.761 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: : exit 0 2026-02-21T01:41:56.762 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:41:56.762 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:41:56.812 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2026-02-21T01:41:56.812 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2026-02-21T01:41:56.865 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:41:56.868 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/test-snaptrim-stats 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2026-02-21T01:41:56.869 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-21T01:41:56.870 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-21T01:41:56.870 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2026-02-21T01:41:56.870 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2026-02-21T01:41:56.871 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none ' 2026-02-21T01:41:56.871 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-21T01:41:56.871 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2026-02-21T01:41:56.872 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2026-02-21T01:41:56.872 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T01:41:56.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2026-02-21T01:41:56.873 INFO:tasks.workunit.client.0.trial106.stderr:TEST_snaptrim_stats_multiple_snaps' 2026-02-21T01:41:56.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2026-02-21T01:41:56.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2026-02-21T01:41:56.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-snaptrim-stats 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:56.874 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:56.875 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:56.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:56.876 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:56.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:56.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:56.878 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:56.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:56.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:56.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:56.880 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:56.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:56.882 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:56.882 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:41:56.883 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:56.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.884 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:56.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:41:56.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:56.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:56.885 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-snaptrim-stats 2026-02-21T01:41:56.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:41:56.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:56.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.153313 2026-02-21T01:41:56.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-snaptrim-stats 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:41:56.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:41:56.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:41:56.892 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:41:56.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:41:56.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:41:56.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:41:56.894 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:41:56.895 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:41:56.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:56.895 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:41:56.896 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:41:56.896 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:41:56.897 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:41:56.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:41:56.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:41:56.899 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:41:56.899 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.899 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:56.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:41:56.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:41:56.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:41:56.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-snaptrim-stats 2026-02-21T01:41:56.902 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:41:56.902 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.902 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:56.903 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.153313 2026-02-21T01:41:56.904 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-snaptrim-stats 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-snaptrim-stats/a 2026-02-21T01:41:56.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:56.928 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:41:56.929 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-21T01:41:56.960 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:41:56.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:41:56.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:56.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:56.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:41:56.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:56.962 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.153313/ceph-mon.a.asok 2026-02-21T01:41:56.963 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:56.963 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.153313/ceph-mon.a.asok config get fsid 2026-02-21T01:41:57.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:57.020 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:57.021 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.153313/ceph-mon.a.asok 2026-02-21T01:41:57.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:41:57.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.153313/ceph-mon.a.asok config get mon_host 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-snaptrim-stats 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-snaptrim-stats/x 2026-02-21T01:41:57.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:41:57.189 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:57.190 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:41:57.209 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2026-02-21T01:41:57.210 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2026-02-21T01:41:57.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T01:41:57.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:41:57.213 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-snaptrim-stats 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-snaptrim-stats/0 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-21T01:41:57.214 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:41:57.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:41:57.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:41:57.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:41:57.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-snaptrim-stats/0 2026-02-21T01:41:57.216 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:41:57.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=76950e3f-3d97-4d4c-a179-ca33e13a2ee1 2026-02-21T01:41:57.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 76950e3f-3d97-4d4c-a179-ca33e13a2ee1' 2026-02-21T01:41:57.217 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 76950e3f-3d97-4d4c-a179-ca33e13a2ee1 2026-02-21T01:41:57.217 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:41:57.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlDZlpikWcDRAAeWF1x+31RCtAAcJb0rS+xA== 2026-02-21T01:41:57.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlDZlpikWcDRAAeWF1x+31RCtAAcJb0rS+xA=="}' 2026-02-21T01:41:57.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 76950e3f-3d97-4d4c-a179-ca33e13a2ee1 -i td/test-snaptrim-stats/0/new.json 2026-02-21T01:41:57.334 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:57.342 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-snaptrim-stats/0/new.json 2026-02-21T01:41:57.343 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBlDZlpikWcDRAAeWF1x+31RCtAAcJb0rS+xA== --osd-uuid 76950e3f-3d97-4d4c-a179-ca33e13a2ee1 2026-02-21T01:41:57.359 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:57.360+0000 7f4f37b85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:57.360 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:57.364+0000 7f4f37b85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:57.361 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:57.364+0000 7f4f37b85a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:57.362 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:57.364+0000 7f4f37b85a40 -1 bdev(0x55f418782c00 td/test-snaptrim-stats/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:41:57.362 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:57.364+0000 7f4f37b85a40 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2026-02-21T01:41:58.140 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2026-02-21T01:41:58.140 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:41:58.141 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:41:58.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:41:58.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:41:58.265 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:41:58.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:41:58.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:41:58.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:41:58.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:41:58.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:41:58.288 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:58.292+0000 7fd518684a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:58.290 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:58.292+0000 7fd518684a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:58.291 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:58.292+0000 7fd518684a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:58.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:58.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:41:58.685 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:58.688+0000 7fd518684a40 -1 Falling back to public interface 2026-02-21T01:41:59.416 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:41:59.420+0000 7fd518684a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:41:59.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:41:59.487 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:41:59.487 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:41:59.488 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:41:59.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:41:59.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:41:59.596 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:00.597 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:00.597 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:00.597 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:00.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:42:00.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:00.598 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:42:00.812 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/426723530,v1:127.0.0.1:6801/426723530] [v2:127.0.0.1:6802/426723530,v1:127.0.0.1:6803/426723530] exists,up 76950e3f-3d97-4d4c-a179-ca33e13a2ee1 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-snaptrim-stats 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-snaptrim-stats/1 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-21T01:42:00.813 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-21T01:42:00.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-snaptrim-stats/1 2026-02-21T01:42:00.815 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:42:00.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2a4e7c89-8588-47c2-b573-b8755a076dcf 2026-02-21T01:42:00.816 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 2a4e7c89-8588-47c2-b573-b8755a076dcf 2026-02-21T01:42:00.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2a4e7c89-8588-47c2-b573-b8755a076dcf' 2026-02-21T01:42:00.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:42:00.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBoDZlpOwisMRAAQx1xjO0tLeOPHY5Ewnkaew== 2026-02-21T01:42:00.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBoDZlpOwisMRAAQx1xjO0tLeOPHY5Ewnkaew=="}' 2026-02-21T01:42:00.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2a4e7c89-8588-47c2-b573-b8755a076dcf -i td/test-snaptrim-stats/1/new.json 2026-02-21T01:42:01.057 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:01.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-snaptrim-stats/1/new.json 2026-02-21T01:42:01.068 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBoDZlpOwisMRAAQx1xjO0tLeOPHY5Ewnkaew== --osd-uuid 2a4e7c89-8588-47c2-b573-b8755a076dcf 2026-02-21T01:42:01.090 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:01.092+0000 7f95fdf4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:01.091 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:01.096+0000 7f95fdf4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:01.092 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:01.096+0000 7f95fdf4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:01.093 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:01.096+0000 7f95fdf4da40 -1 bdev(0x564385b70c00 td/test-snaptrim-stats/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:42:01.093 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:01.096+0000 7f95fdf4da40 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2026-02-21T01:42:01.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2026-02-21T01:42:01.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:42:01.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:42:01.760 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:42:01.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:42:02.057 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:42:02.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:42:02.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:02.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:42:02.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:42:02.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:42:02.080 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:02.084+0000 7f2787df0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:02.082 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:02.084+0000 7f2787df0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:02.083 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:02.084+0000 7f2787df0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:02.262 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:42:02.263 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:42:02.263 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:02.263 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:42:02.469 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:02.472+0000 7f2787df0a40 -1 Falling back to public interface 2026-02-21T01:42:02.471 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:03.039 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:03.040+0000 7f2787df0a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:42:03.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:03.473 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:03.474 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:03.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:42:03.474 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:03.475 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:42:03.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:04.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:04.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:04.678 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:04.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:42:04.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:04.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:42:04.894 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/877397842,v1:127.0.0.1:6811/877397842] [v2:127.0.0.1:6812/877397842,v1:127.0.0.1:6813/877397842] exists,up 2a4e7c89-8588-47c2-b573-b8755a076dcf 2026-02-21T01:42:04.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:42:04.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:42:04.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:42:04.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T01:42:04.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:04.898 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-snaptrim-stats 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-snaptrim-stats/2 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:42:04.899 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:42:04.900 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-snaptrim-stats/2 2026-02-21T01:42:04.901 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:42:04.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b727d5ce-dd20-4b52-a810-4547369a2941 2026-02-21T01:42:04.902 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b727d5ce-dd20-4b52-a810-4547369a2941' 2026-02-21T01:42:04.902 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 b727d5ce-dd20-4b52-a810-4547369a2941 2026-02-21T01:42:04.903 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:42:04.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBsDZlpLiSENhAAxYGvF6an1HJXSmLDpCZ59A== 2026-02-21T01:42:04.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBsDZlpLiSENhAAxYGvF6an1HJXSmLDpCZ59A=="}' 2026-02-21T01:42:04.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b727d5ce-dd20-4b52-a810-4547369a2941 -i td/test-snaptrim-stats/2/new.json 2026-02-21T01:42:05.124 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:05.133 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-snaptrim-stats/2/new.json 2026-02-21T01:42:05.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBsDZlpLiSENhAAxYGvF6an1HJXSmLDpCZ59A== --osd-uuid b727d5ce-dd20-4b52-a810-4547369a2941 2026-02-21T01:42:05.150 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:05.151+0000 7fd58e68ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:05.151 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:05.151+0000 7fd58e68ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:05.152 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:05.155+0000 7fd58e68ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:05.153 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:05.155+0000 7fd58e68ba40 -1 bdev(0x55b2d0446c00 td/test-snaptrim-stats/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:42:05.153 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:05.155+0000 7fd58e68ba40 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2026-02-21T01:42:05.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2026-02-21T01:42:05.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:42:05.832 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:42:05.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:42:05.833 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:42:06.118 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:42:06.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:42:06.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:06.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:42:06.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:42:06.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:42:06.142 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:06.143+0000 7fa407eaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:06.144 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:06.147+0000 7fa407eaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:06.145 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:06.147+0000 7fa407eaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:42:06.332 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:42:06.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:06.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:42:06.517 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:06.519+0000 7fa407eaea40 -1 Falling back to public interface 2026-02-21T01:42:06.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:07.151 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:07.151+0000 7fa407eaea40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:42:07.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:07.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:07.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:42:07.550 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:07.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:07.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:42:07.765 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:08.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:08.766 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:08.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:08.766 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:42:08.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:08.767 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:42:08.971 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3213009665,v1:127.0.0.1:6819/3213009665] [v2:127.0.0.1:6820/3213009665,v1:127.0.0.1:6821/3213009665] exists,up b727d5ce-dd20-4b52-a810-4547369a2941 2026-02-21T01:42:08.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:42:08.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:42:08.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:42:08.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2026-02-21T01:42:09.270 INFO:tasks.workunit.client.0.trial106.stderr:noscrub is set 2026-02-21T01:42:09.282 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2026-02-21T01:42:09.574 INFO:tasks.workunit.client.0.trial106.stderr:nodeep-scrub is set 2026-02-21T01:42:09.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2026-02-21T01:42:09.586 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 8 8 2026-02-21T01:42:09.831 INFO:tasks.workunit.client.0.trial106.stderr:pool 'test' already exists 2026-02-21T01:42:09.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:42:10.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2026-02-21T01:42:10.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:42:10.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:42:10.844 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:42:10.845 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:42:10.845 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:42:10.845 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:42:10.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:42:10.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:42:10.846 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:42:10.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:42:10.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:42:10.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:42:10.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:42:10.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:42:10.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:42:11.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:42:11.159 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:42:11.159 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:42:11.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:42:11.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:11.159 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:42:11.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-21T01:42:11.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-21T01:42:11.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-21T01:42:11.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:11.236 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:42:11.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-21T01:42:11.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-21T01:42:11.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-21T01:42:11.317 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:11.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:42:11.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-21T01:42:11.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-21T01:42:11.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-64424509443' 2026-02-21T01:42:11.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:11.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-21T01:42:11.393 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:11.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:42:11.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-21T01:42:11.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:11.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-21T01:42:11.396 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836485 2026-02-21T01:42:11.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-21T01:42:11.396 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:42:11.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-21T01:42:11.609 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:11.610 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-21T01:42:11.610 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:11.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:42:11.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-21T01:42:11.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:11.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-21T01:42:11.614 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672964 2026-02-21T01:42:11.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-21T01:42:11.614 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:42:11.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-21T01:42:11.826 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:11.827 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-21T01:42:11.827 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:11.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:42:11.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-21T01:42:11.828 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:11.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-21T01:42:11.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-21T01:42:11.830 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 64424509443 2026-02-21T01:42:11.830 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:42:12.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-21T01:42:12.054 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:42:12.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:12.055 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:12.345 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-21T01:42:12.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:42:12.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:42:12.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:42:12.346 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:42:12.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:42:12.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:42:12.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:42:12.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-21T01:42:12.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:42:12.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:12.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:12.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-21T01:42:12.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:42:12.843 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:42:12.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2026-02-21T01:42:12.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2026-02-21T01:42:12.844 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2026-02-21T01:42:13.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2026-02-21T01:42:13.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2026-02-21T01:42:13.059 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2026-02-21T01:42:13.060 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:42:13.060 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:42:13.060 INFO:tasks.workunit.client.0.trial106.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.2498e-05 s, 96.4 MB/s 2026-02-21T01:42:13.060 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2026-02-21T01:42:13.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2026-02-21T01:42:13.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2026-02-21T01:42:13.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2026-02-21T01:42:13.125 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.125 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2026-02-21T01:42:13.148 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.148 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2026-02-21T01:42:13.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2026-02-21T01:42:13.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2026-02-21T01:42:13.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2026-02-21T01:42:13.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2026-02-21T01:42:13.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2026-02-21T01:42:13.290 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2026-02-21T01:42:13.291 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2026-02-21T01:42:13.291 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2026-02-21T01:42:13.364 INFO:tasks.workunit.client.0.trial106.stdout:created pool test snap snap1 2026-02-21T01:42:13.367 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2026-02-21T01:42:13.367 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2026-02-21T01:42:13.367 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:42:13.368 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:42:13.368 INFO:tasks.workunit.client.0.trial106.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.7524e-05 s, 60.7 MB/s 2026-02-21T01:42:13.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2026-02-21T01:42:13.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.369 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2026-02-21T01:42:13.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2026-02-21T01:42:13.410 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2026-02-21T01:42:13.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2026-02-21T01:42:13.459 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2026-02-21T01:42:13.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.484 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2026-02-21T01:42:13.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2026-02-21T01:42:13.535 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.535 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2026-02-21T01:42:13.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2026-02-21T01:42:13.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2026-02-21T01:42:13.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2026-02-21T01:42:13.610 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2026-02-21T01:42:13.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2026-02-21T01:42:13.683 INFO:tasks.workunit.client.0.trial106.stdout:removed pool test snap snap1 2026-02-21T01:42:13.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2026-02-21T01:42:13.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:42:13.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:42:13.686 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:42:13.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:42:13.687 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:42:13.687 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:42:13.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:42:13.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:42:13.687 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:42:13.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:42:13.783 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:42:13.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:42:13.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:42:13.784 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:42:13.784 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:42:13.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:42:13.998 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:42:13.999 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:42:13.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:42:13.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:13.999 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:42:14.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-21T01:42:14.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-21T01:42:14.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-21T01:42:14.077 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:14.078 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:42:14.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-21T01:42:14.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-21T01:42:14.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2026-02-21T01:42:14.155 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:14.155 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:42:14.231 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509446 2026-02-21T01:42:14.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509446 2026-02-21T01:42:14.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509446' 2026-02-21T01:42:14.232 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:14.232 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-21T01:42:14.233 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:14.233 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:42:14.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-21T01:42:14.234 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:14.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-21T01:42:14.236 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-21T01:42:14.236 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836487 2026-02-21T01:42:14.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:42:14.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836487 2026-02-21T01:42:14.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:42:15.458 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:42:15.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:42:15.665 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-21T01:42:15.666 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:15.666 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-21T01:42:15.667 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:15.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:42:15.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-21T01:42:15.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:15.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-21T01:42:15.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-21T01:42:15.670 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672966 2026-02-21T01:42:15.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:42:15.876 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672966 2026-02-21T01:42:15.877 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:15.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509446 2026-02-21T01:42:15.878 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:15.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:42:15.879 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509446 2026-02-21T01:42:15.880 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:15.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509446 2026-02-21T01:42:15.881 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2026-02-21T01:42:15.881 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 64424509446 2026-02-21T01:42:15.881 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:42:16.100 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509446 -lt 64424509446 2026-02-21T01:42:16.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:42:16.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:16.101 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:16.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-21T01:42:16.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:42:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:42:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:42:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:42:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:42:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:42:16.382 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:42:16.584 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-21T01:42:16.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:42:16.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:16.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:16.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-21T01:42:16.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:42:16.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:42:16.858 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2026-02-21T01:42:26.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2026-02-21T01:42:26.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2026-02-21T01:42:26.860 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2026-02-21T01:42:26.861 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2026-02-21T01:42:26.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:26.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2026-02-21T01:42:26.864 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2026-02-21T01:42:26.864 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:26.949 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2026-02-21T01:42:26.950 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2026-02-21T01:42:26.951 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:26.952 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2026-02-21T01:42:26.952 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:27.032 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.00150562 2026-02-21T01:42:27.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.00150562 2026-02-21T01:42:27.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:27.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2026-02-21T01:42:27.034 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2026-02-21T01:42:27.034 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:27.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2026-02-21T01:42:27.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2026-02-21T01:42:27.116 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:27.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2026-02-21T01:42:27.117 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:27.194 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .00150562 + 0.00203486 2026-02-21T01:42:27.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.00354048 2026-02-21T01:42:27.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:27.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2026-02-21T01:42:27.196 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2026-02-21T01:42:27.196 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:27.277 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2026-02-21T01:42:27.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2026-02-21T01:42:27.279 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:27.280 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2026-02-21T01:42:27.280 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:27.359 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .00354048 + 0.000333321 2026-02-21T01:42:27.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.003873801 2026-02-21T01:42:27.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:27.360 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2026-02-21T01:42:27.361 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2026-02-21T01:42:27.361 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:27.439 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2026-02-21T01:42:27.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2026-02-21T01:42:27.441 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:27.442 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2026-02-21T01:42:27.442 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:27.519 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .003873801 + 0.003086739 2026-02-21T01:42:27.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.006960540 2026-02-21T01:42:27.521 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:27.522 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2026-02-21T01:42:27.522 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2026-02-21T01:42:27.522 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:27.602 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2026-02-21T01:42:27.603 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2026-02-21T01:42:27.604 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:27.605 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2026-02-21T01:42:27.605 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:27.682 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .006960540 + 0.001470296 2026-02-21T01:42:27.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.008430836 2026-02-21T01:42:27.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:27.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2026-02-21T01:42:27.683 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2026-02-21T01:42:27.683 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:27.764 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2026-02-21T01:42:27.764 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2026-02-21T01:42:27.765 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:27.766 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2026-02-21T01:42:27.766 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:27.849 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .008430836 + 0.001766839 2026-02-21T01:42:27.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.010197675 2026-02-21T01:42:27.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:27.850 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2026-02-21T01:42:27.851 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2026-02-21T01:42:27.851 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:27.934 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2026-02-21T01:42:27.934 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2026-02-21T01:42:27.935 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:27.936 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2026-02-21T01:42:27.936 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:28.014 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .010197675 + 0.002901458 2026-02-21T01:42:28.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.013099133 2026-02-21T01:42:28.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:42:28.015 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2026-02-21T01:42:28.016 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2026-02-21T01:42:28.016 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2026-02-21T01:42:28.097 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2026-02-21T01:42:28.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2026-02-21T01:42:28.099 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2026-02-21T01:42:28.100 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2026-02-21T01:42:28.100 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2026-02-21T01:42:28.178 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .013099133 + 0.002711581 2026-02-21T01:42:28.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.015810714 2026-02-21T01:42:28.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2026-02-21T01:42:28.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '.015810714 > 0.0' 2026-02-21T01:42:28.179 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2026-02-21T01:42:28.180 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:28.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2026-02-21T01:42:28.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:42:28.180 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:42:28.180 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:42:28.181 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:42:28.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:42:28.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:42:28.181 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:42:28.492 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:42:28.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:42:28.493 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:42:28.494 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:42:28.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:42:28.496 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:42:28.496 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:42:28.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.497 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:42:28.498 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:42:28.498 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.499 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:42:28.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:42:28.500 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:42:28.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:42:28.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:42:28.539 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:42:28.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:42:28.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:42:28.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:42:28.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:42:28.543 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:42:28.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:42:28.544 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:42:28.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:42:28.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:42:28.545 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:42:28.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:42:28.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:42:28.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:42:28.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:42:28.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:42:28.550 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:42:28.550 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-snaptrim-stats 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:42:28.552 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:42:28.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:42:28.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:42:28.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:42:28.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:42:28.555 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:42:28.555 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:42:28.556 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:42:28.556 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:42:28.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:42:28.558 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:42:28.558 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:42:28.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:42:28.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:42:28.560 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.561 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:42:28.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:42:28.562 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:42:28.563 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:42:28.563 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.563 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.563 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:42:28.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:42:28.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:42:28.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-snaptrim-stats 2026-02-21T01:42:28.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:42:28.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.566 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.153313 2026-02-21T01:42:28.568 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:42:28.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/test-snaptrim-stats 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:42:28.569 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:42:28.570 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:42:28.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:42:28.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:42:28.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:42:28.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:42:28.571 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:42:28.572 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:42:28.572 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:42:28.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:42:28.573 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:42:28.574 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:42:28.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:42:28.575 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:42:28.576 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:42:28.577 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:42:28.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:42:28.578 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:42:28.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:42:28.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:42:28.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:42:28.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:42:28.581 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/test-snaptrim-stats 2026-02-21T01:42:28.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:42:28.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.582 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.582 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.153313 2026-02-21T01:42:28.584 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:42:28.584 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/test-snaptrim-stats/a 2026-02-21T01:42:28.585 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2026-02-21T01:42:28.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:42:28.607 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:28.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:28.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:28.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.608 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:28.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-21T01:42:28.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:42:28.636 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:42:28.636 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:42:28.637 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:42:28.637 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:42:28.638 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:42:28.638 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:42:28.638 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:42:28.638 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:42:28.638 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:42:28.638 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.639 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.639 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.153313/ceph-mon.a.asok 2026-02-21T01:42:28.639 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:42:28.639 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.153313/ceph-mon.a.asok config get fsid 2026-02-21T01:42:28.695 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:42:28.695 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:42:28.695 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:42:28.695 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:42:28.695 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:42:28.696 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:42:28.696 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:42:28.696 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:42:28.697 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:42:28.697 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.697 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.697 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.153313/ceph-mon.a.asok 2026-02-21T01:42:28.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:42:28.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.153313/ceph-mon.a.asok config get mon_host 2026-02-21T01:42:28.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2026-02-21T01:42:28.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.752 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:42:28.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:42:28.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:42:28.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/test-snaptrim-stats/x 2026-02-21T01:42:28.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.865 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:28.866 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:42:28.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:42:28.885 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2026-02-21T01:42:28.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2026-02-21T01:42:28.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T01:42:28.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-snaptrim-stats 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-snaptrim-stats/0 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2026-02-21T01:42:28.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:42:28.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:28.892 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-snaptrim-stats/0 2026-02-21T01:42:28.893 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:42:28.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bee79183-20a3-4ba6-8027-f15ae5c1c7fd 2026-02-21T01:42:28.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 bee79183-20a3-4ba6-8027-f15ae5c1c7fd' 2026-02-21T01:42:28.893 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 bee79183-20a3-4ba6-8027-f15ae5c1c7fd 2026-02-21T01:42:28.894 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:42:28.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCEDZlpKXvzNRAAKaibV2FQgNhc8YWZG3NrhA== 2026-02-21T01:42:28.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCEDZlpKXvzNRAAKaibV2FQgNhc8YWZG3NrhA=="}' 2026-02-21T01:42:28.901 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bee79183-20a3-4ba6-8027-f15ae5c1c7fd -i td/test-snaptrim-stats/0/new.json 2026-02-21T01:42:29.010 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:42:29.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-snaptrim-stats/0/new.json 2026-02-21T01:42:29.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCEDZlpKXvzNRAAKaibV2FQgNhc8YWZG3NrhA== --osd-uuid bee79183-20a3-4ba6-8027-f15ae5c1c7fd 2026-02-21T01:42:29.035 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.034+0000 7f0a83c7ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:29.036 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.038+0000 7f0a83c7ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:29.037 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.038+0000 7f0a83c7ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:29.037 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.038+0000 7f0a83c7ba40 -1 bdev(0x558bae708c00 td/test-snaptrim-stats/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:42:29.037 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.038+0000 7f0a83c7ba40 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2026-02-21T01:42:29.691 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2026-02-21T01:42:29.691 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:42:29.692 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:42:29.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:42:29.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:42:29.805 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:42:29.806 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:42:29.806 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:29.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:42:29.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:42:29.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:42:29.821 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.822+0000 7f89ea7a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:29.822 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.822+0000 7f89ea7a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:29.823 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:29.822+0000 7f89ea7a9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:29.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:42:29.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:42:29.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:42:29.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:42:29.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:42:29.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:29.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:42:29.920 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:42:29.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:29.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:42:30.027 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:30.210 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:30.210+0000 7f89ea7a9a40 -1 Falling back to public interface 2026-02-21T01:42:30.932 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:30.930+0000 7f89ea7a9a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:42:31.028 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:31.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:31.029 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:31.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:42:31.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:31.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:42:31.170 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:32.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:32.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:32.171 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:32.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:42:32.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:32.171 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:42:32.385 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2642839898,v1:127.0.0.1:6801/2642839898] [v2:127.0.0.1:6802/2642839898,v1:127.0.0.1:6803/2642839898] exists,up bee79183-20a3-4ba6-8027-f15ae5c1c7fd 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-snaptrim-stats 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-snaptrim-stats/1 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:32.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:32.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-snaptrim-stats/1 2026-02-21T01:42:32.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:42:32.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7f1e3a17-96b3-4c4b-bb04-3efbcb484c9f 2026-02-21T01:42:32.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7f1e3a17-96b3-4c4b-bb04-3efbcb484c9f' 2026-02-21T01:42:32.390 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 7f1e3a17-96b3-4c4b-bb04-3efbcb484c9f 2026-02-21T01:42:32.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:42:32.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCIDZlpS7wyGBAAXDdw/hmj/CqN3d+Xb5h4hA== 2026-02-21T01:42:32.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCIDZlpS7wyGBAAXDdw/hmj/CqN3d+Xb5h4hA=="}' 2026-02-21T01:42:32.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7f1e3a17-96b3-4c4b-bb04-3efbcb484c9f -i td/test-snaptrim-stats/1/new.json 2026-02-21T01:42:32.629 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:32.638 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-snaptrim-stats/1/new.json 2026-02-21T01:42:32.639 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCIDZlpS7wyGBAAXDdw/hmj/CqN3d+Xb5h4hA== --osd-uuid 7f1e3a17-96b3-4c4b-bb04-3efbcb484c9f 2026-02-21T01:42:32.657 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:32.658+0000 7f93509d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:32.659 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:32.658+0000 7f93509d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:32.660 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:32.658+0000 7f93509d3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:32.660 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:32.658+0000 7f93509d3a40 -1 bdev(0x564629edec00 td/test-snaptrim-stats/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:42:32.660 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:32.662+0000 7f93509d3a40 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2026-02-21T01:42:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2026-02-21T01:42:33.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:42:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:42:33.332 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:42:33.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:42:33.612 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:42:33.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:42:33.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:33.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:42:33.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:42:33.613 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:42:33.629 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:33.630+0000 7f7eb19a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:33.630 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:33.630+0000 7f7eb19a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:33.631 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:33.630+0000 7f7eb19a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:33.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:42:33.819 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:42:33.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:33.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:42:33.986 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:33.986+0000 7f7eb19a6a40 -1 Falling back to public interface 2026-02-21T01:42:34.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:34.643 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:34.642+0000 7f7eb19a6a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:42:35.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:35.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:35.036 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:42:35.036 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:35.037 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:35.037 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:42:35.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:36.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:36.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:36.243 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:36.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:42:36.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:36.244 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:42:36.451 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3230357067,v1:127.0.0.1:6811/3230357067] [v2:127.0.0.1:6812/3230357067,v1:127.0.0.1:6813/3230357067] exists,up 7f1e3a17-96b3-4c4b-bb04-3efbcb484c9f 2026-02-21T01:42:36.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:42:36.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:42:36.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:42:36.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T01:42:36.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:36.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/test-snaptrim-stats 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/test-snaptrim-stats/2 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:42:36.455 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:36.456 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/test-snaptrim-stats/2 2026-02-21T01:42:36.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:42:36.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fcebc0b8-8d26-4c37-8ccd-76ecd84fa86b 2026-02-21T01:42:36.457 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 fcebc0b8-8d26-4c37-8ccd-76ecd84fa86b 2026-02-21T01:42:36.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fcebc0b8-8d26-4c37-8ccd-76ecd84fa86b' 2026-02-21T01:42:36.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:42:36.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMDZlpTr0RHBAA5vblaCTUyUndStcC17gE5g== 2026-02-21T01:42:36.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMDZlpTr0RHBAA5vblaCTUyUndStcC17gE5g=="}' 2026-02-21T01:42:36.467 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fcebc0b8-8d26-4c37-8ccd-76ecd84fa86b -i td/test-snaptrim-stats/2/new.json 2026-02-21T01:42:36.683 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:36.693 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/test-snaptrim-stats/2/new.json 2026-02-21T01:42:36.694 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCMDZlpTr0RHBAA5vblaCTUyUndStcC17gE5g== --osd-uuid fcebc0b8-8d26-4c37-8ccd-76ecd84fa86b 2026-02-21T01:42:36.710 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:36.710+0000 7f792de86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:36.711 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:36.710+0000 7f792de86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:36.712 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:36.710+0000 7f792de86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:36.712 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:36.714+0000 7f792de86a40 -1 bdev(0x55fd3e89ec00 td/test-snaptrim-stats/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:42:36.713 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:36.714+0000 7f792de86a40 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2026-02-21T01:42:37.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2026-02-21T01:42:37.324 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:42:37.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:42:37.325 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:42:37.325 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:42:37.616 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:42:37.617 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:42:37.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=9ed68900-0334-4fec-b255-6738c2b07996 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.153313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2026-02-21T01:42:37.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:42:37.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:42:37.618 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:42:37.640 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:37.638+0000 7f9c94f4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:37.641 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:37.642+0000 7f9c94f4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:37.642 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:37.642+0000 7f9c94f4ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:42:37.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:42:37.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:42:37.830 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:37.831 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:42:37.998 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:37.998+0000 7f9c94f4ca40 -1 Falling back to public interface 2026-02-21T01:42:38.038 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:38.652 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:42:38.654+0000 7f9c94f4ca40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:42:39.040 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:39.040 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:39.040 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:42:39.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:42:39.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:39.041 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:42:39.255 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:42:40.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:42:40.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:42:40.257 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:42:40.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:42:40.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:42:40.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:42:40.468 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1764429407,v1:127.0.0.1:6819/1764429407] [v2:127.0.0.1:6820/1764429407,v1:127.0.0.1:6821/1764429407] exists,up fcebc0b8-8d26-4c37-8ccd-76ecd84fa86b 2026-02-21T01:42:40.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:42:40.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:42:40.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:42:40.469 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2026-02-21T01:42:40.760 INFO:tasks.workunit.client.0.trial106.stderr:noscrub is set 2026-02-21T01:42:40.771 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2026-02-21T01:42:41.064 INFO:tasks.workunit.client.0.trial106.stderr:nodeep-scrub is set 2026-02-21T01:42:41.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2026-02-21T01:42:41.076 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 8 8 2026-02-21T01:42:41.321 INFO:tasks.workunit.client.0.trial106.stderr:pool 'test' already exists 2026-02-21T01:42:41.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T01:42:42.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2026-02-21T01:42:42.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:42:42.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:42:42.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:42:42.334 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:42:42.335 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:42:42.335 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:42:42.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:42:42.335 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:42:42.336 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:42:42.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:42:42.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:42:42.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:42:42.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:42:42.442 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:42:42.442 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:42:42.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:42:42.653 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:42:42.653 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:42:42.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:42:42.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:42.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:42:42.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-21T01:42:42.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-21T01:42:42.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-21T01:42:42.729 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:42.729 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:42:42.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-21T01:42:42.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-21T01:42:42.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-21T01:42:42.807 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:42.807 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:42:42.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-21T01:42:42.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-21T01:42:42.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-60129542147' 2026-02-21T01:42:42.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:42.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-21T01:42:42.887 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:42.887 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:42:42.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-21T01:42:42.888 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:42.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-21T01:42:42.890 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-21T01:42:42.890 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836485 2026-02-21T01:42:42.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:42:43.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836485 2026-02-21T01:42:43.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:42:44.105 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:42:44.106 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:42:44.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-21T01:42:44.308 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:44.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-21T01:42:44.309 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:44.310 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:42:44.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-21T01:42:44.311 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:44.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-21T01:42:44.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-21T01:42:44.312 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672964 2026-02-21T01:42:44.312 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:42:44.535 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-21T01:42:44.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:44.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-21T01:42:44.536 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:44.537 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:42:44.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-21T01:42:44.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:44.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-21T01:42:44.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-21T01:42:44.540 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542147 2026-02-21T01:42:44.540 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:42:44.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-21T01:42:44.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:42:44.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:44.762 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:45.044 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:42:45.045 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:42:45.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-21T01:42:45.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:42:45.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:45.260 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:45.533 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-21T01:42:45.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:42:45.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:42:45.534 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2026-02-21T01:42:45.534 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2026-02-21T01:42:45.534 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2026-02-21T01:42:45.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2026-02-21T01:42:45.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2026-02-21T01:42:45.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2026-02-21T01:42:45.742 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:42:45.742 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:42:45.742 INFO:tasks.workunit.client.0.trial106.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 4.2277e-05 s, 96.9 MB/s 2026-02-21T01:42:45.743 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2026-02-21T01:42:45.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2026-02-21T01:42:45.762 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.763 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2026-02-21T01:42:45.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.787 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2026-02-21T01:42:45.810 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.810 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2026-02-21T01:42:45.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2026-02-21T01:42:45.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.859 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2026-02-21T01:42:45.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.884 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2026-02-21T01:42:45.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2026-02-21T01:42:45.927 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.927 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2026-02-21T01:42:45.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:45.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2026-02-21T01:42:45.975 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2026-02-21T01:42:45.976 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2026-02-21T01:42:45.976 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2026-02-21T01:42:45.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-21T01:42:45.977 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2026-02-21T01:42:46.046 INFO:tasks.workunit.client.0.trial106.stdout:created pool test snap snap1 2026-02-21T01:42:46.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2026-02-21T01:42:46.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2026-02-21T01:42:46.049 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:42:46.049 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:42:46.049 INFO:tasks.workunit.client.0.trial106.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.1042e-05 s, 67.1 MB/s 2026-02-21T01:42:46.049 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2026-02-21T01:42:46.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.050 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2026-02-21T01:42:46.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2026-02-21T01:42:46.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.094 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2026-02-21T01:42:46.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2026-02-21T01:42:46.133 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.133 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2026-02-21T01:42:46.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2026-02-21T01:42:46.182 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.183 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2026-02-21T01:42:46.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.207 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2026-02-21T01:42:46.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2026-02-21T01:42:46.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2026-02-21T01:42:46.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2026-02-21T01:42:46.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-21T01:42:46.271 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2026-02-21T01:42:46.343 INFO:tasks.workunit.client.0.trial106.stdout:created pool test snap snap2 2026-02-21T01:42:46.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2026-02-21T01:42:46.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2026-02-21T01:42:46.347 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records in 2026-02-21T01:42:46.347 INFO:tasks.workunit.client.0.trial106.stderr:1+0 records out 2026-02-21T01:42:46.347 INFO:tasks.workunit.client.0.trial106.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.3566e-05 s, 64.4 MB/s 2026-02-21T01:42:46.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2026-02-21T01:42:46.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.348 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2026-02-21T01:42:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2026-02-21T01:42:46.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2026-02-21T01:42:46.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.417 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2026-02-21T01:42:46.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.441 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2026-02-21T01:42:46.465 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.466 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2026-02-21T01:42:46.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.490 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2026-02-21T01:42:46.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2026-02-21T01:42:46.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2026-02-21T01:42:46.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2026-02-21T01:42:46.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2026-02-21T01:42:46.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2026-02-21T01:42:46.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2026-02-21T01:42:46.592 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2026-02-21T01:42:46.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-21T01:42:46.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2026-02-21T01:42:46.662 INFO:tasks.workunit.client.0.trial106.stdout:removed pool test snap snap1 2026-02-21T01:42:46.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2026-02-21T01:42:46.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:42:46.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:42:46.664 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:42:46.664 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:42:46.665 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:42:46.665 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:42:46.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:42:46.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:42:46.665 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:42:46.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:42:46.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:42:46.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:42:46.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:42:46.777 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:42:46.778 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:42:46.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:42:46.993 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:42:46.993 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:42:46.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:42:46.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:46.994 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:42:47.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-21T01:42:47.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-21T01:42:47.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-21T01:42:47.071 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:47.072 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:42:47.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-21T01:42:47.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-21T01:42:47.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672967' 2026-02-21T01:42:47.149 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:42:47.149 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:42:47.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-21T01:42:47.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-21T01:42:47.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672967 2-60129542150' 2026-02-21T01:42:47.237 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:47.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-21T01:42:47.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:47.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:42:47.239 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-21T01:42:47.239 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:47.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-21T01:42:47.241 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-21T01:42:47.241 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836487 2026-02-21T01:42:47.241 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:42:47.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-21T01:42:47.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:47.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-21T01:42:47.449 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:47.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:42:47.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-21T01:42:47.451 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:47.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-21T01:42:47.452 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-21T01:42:47.453 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672967 2026-02-21T01:42:47.453 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:42:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672967 2026-02-21T01:42:47.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:42:47.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-21T01:42:47.669 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:42:47.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:42:47.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-21T01:42:47.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:42:47.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-21T01:42:47.673 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542150 2026-02-21T01:42:47.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-21T01:42:47.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:42:47.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542150 2026-02-21T01:42:47.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:42:48.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:42:48.881 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:42:49.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542150 2026-02-21T01:42:49.098 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:42:50.099 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T01:42:50.100 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:42:50.307 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-21T01:42:50.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:42:50.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:50.308 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:42:50.579 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:42:50.580 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:42:50.780 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-21T01:42:50.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:42:50.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:42:50.780 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:42:51.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-21T01:42:51.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:42:51.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:42:51.065 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2026-02-21T01:43:01.066 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2026-02-21T01:43:01.067 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2026-02-21T01:43:01.067 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2026-02-21T01:43:01.069 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2026-02-21T01:43:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:01.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2026-02-21T01:43:01.072 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-21T01:43:01.072 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:01.152 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2026-02-21T01:43:01.153 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2026-02-21T01:43:01.153 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:01.154 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-21T01:43:01.154 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:01.237 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.002192405 2026-02-21T01:43:01.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.002192405 2026-02-21T01:43:01.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:01.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2026-02-21T01:43:01.239 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-21T01:43:01.239 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:01.317 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2026-02-21T01:43:01.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-21T01:43:01.319 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:01.319 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-21T01:43:01.320 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:01.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .002192405 + 0.002030699 2026-02-21T01:43:01.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.004223104 2026-02-21T01:43:01.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:01.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2026-02-21T01:43:01.399 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-21T01:43:01.399 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:01.481 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2026-02-21T01:43:01.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-21T01:43:01.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:01.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-21T01:43:01.483 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:01.563 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .004223104 + 0.000400484 2026-02-21T01:43:01.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.004623588 2026-02-21T01:43:01.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:01.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2026-02-21T01:43:01.565 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-21T01:43:01.565 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:01.644 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2026-02-21T01:43:01.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2026-02-21T01:43:01.645 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:01.646 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-21T01:43:01.646 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:01.725 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .004623588 + 0.00220587 2026-02-21T01:43:01.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.006829458 2026-02-21T01:43:01.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:01.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2026-02-21T01:43:01.726 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-21T01:43:01.727 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:01.808 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2026-02-21T01:43:01.808 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2026-02-21T01:43:01.809 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:01.810 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-21T01:43:01.810 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:01.890 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .006829458 + 0.003666776 2026-02-21T01:43:01.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.010496234 2026-02-21T01:43:01.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:01.891 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2026-02-21T01:43:01.892 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-21T01:43:01.892 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:01.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2026-02-21T01:43:01.972 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-21T01:43:01.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:01.973 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-21T01:43:01.974 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:02.052 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .010496234 + 0.00181829 2026-02-21T01:43:02.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.012314524 2026-02-21T01:43:02.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:02.054 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2026-02-21T01:43:02.054 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-21T01:43:02.054 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:02.135 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2026-02-21T01:43:02.136 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-21T01:43:02.137 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:02.137 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-21T01:43:02.138 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:02.217 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .012314524 + 0.002463021 2026-02-21T01:43:02.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.014777545 2026-02-21T01:43:02.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:02.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2026-02-21T01:43:02.219 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-21T01:43:02.219 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:02.296 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2026-02-21T01:43:02.297 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2026-02-21T01:43:02.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:02.298 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-21T01:43:02.299 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:02.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .014777545 + 0.004016696 2026-02-21T01:43:02.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.018794241 2026-02-21T01:43:02.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2026-02-21T01:43:02.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '.018794241 > 0.0' 2026-02-21T01:43:02.379 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:02.379 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:02.379 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2026-02-21T01:43:02.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2026-02-21T01:43:02.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2026-02-21T01:43:02.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2026-02-21T01:43:02.448 INFO:tasks.workunit.client.0.trial106.stdout:removed pool test snap snap2 2026-02-21T01:43:02.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2026-02-21T01:43:02.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T01:43:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T01:43:02.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T01:43:02.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T01:43:02.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T01:43:02.452 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T01:43:02.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T01:43:02.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T01:43:02.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T01:43:02.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T01:43:02.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T01:43:02.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T01:43:02.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T01:43:02.547 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-21T01:43:02.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-21T01:43:02.756 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-21T01:43:02.757 INFO:tasks.workunit.client.0.trial106.stderr:1 2026-02-21T01:43:02.757 INFO:tasks.workunit.client.0.trial106.stderr:2' 2026-02-21T01:43:02.757 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-21T01:43:02.757 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:43:02.757 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-21T01:43:02.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-21T01:43:02.837 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-21T01:43:02.837 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-21T01:43:02.837 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:43:02.837 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-21T01:43:02.913 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672972 2026-02-21T01:43:02.913 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672972 2026-02-21T01:43:02.913 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2026-02-21T01:43:02.913 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-21T01:43:02.914 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-21T01:43:02.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-21T01:43:02.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-21T01:43:02.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-60129542155' 2026-02-21T01:43:02.993 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:43:02.993 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-21T01:43:02.993 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:43:02.995 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-21T01:43:02.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-21T01:43:02.995 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:43:02.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-21T01:43:02.997 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-21T01:43:02.997 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.0 seq 21474836493 2026-02-21T01:43:02.997 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:43:03.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836493 2026-02-21T01:43:03.214 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-21T01:43:04.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T01:43:04.216 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T01:43:04.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-21T01:43:04.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:43:04.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672972 2026-02-21T01:43:04.429 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:43:04.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-21T01:43:04.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672972 2026-02-21T01:43:04.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:43:04.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672972 2026-02-21T01:43:04.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2026-02-21T01:43:04.433 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.1 seq 42949672972 2026-02-21T01:43:04.433 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T01:43:04.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672972 -lt 42949672972 2026-02-21T01:43:04.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-21T01:43:04.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-21T01:43:04.651 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-21T01:43:04.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-21T01:43:04.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-21T01:43:04.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-21T01:43:04.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-21T01:43:04.654 INFO:tasks.workunit.client.0.trial106.stdout:waiting osd.2 seq 60129542155 2026-02-21T01:43:04.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-21T01:43:04.654 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T01:43:04.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-21T01:43:04.867 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T01:43:04.867 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:43:04.867 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:43:05.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-21T01:43:05.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T01:43:05.137 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T01:43:05.137 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T01:43:05.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T01:43:05.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T01:43:05.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T01:43:05.138 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T01:43:05.340 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-21T01:43:05.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T01:43:05.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T01:43:05.340 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T01:43:05.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-21T01:43:05.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T01:43:05.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T01:43:05.630 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2026-02-21T01:43:15.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2026-02-21T01:43:15.631 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2026-02-21T01:43:15.632 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2026-02-21T01:43:15.634 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2026-02-21T01:43:15.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:15.635 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2026-02-21T01:43:15.636 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-21T01:43:15.637 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:15.717 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2026-02-21T01:43:15.718 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2026-02-21T01:43:15.718 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:15.719 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2026-02-21T01:43:15.719 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:15.796 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.002255054 2026-02-21T01:43:15.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.002255054 2026-02-21T01:43:15.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:15.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2026-02-21T01:43:15.798 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-21T01:43:15.798 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:15.877 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2026-02-21T01:43:15.878 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-21T01:43:15.878 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:15.879 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2026-02-21T01:43:15.879 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:15.961 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .002255054 + 0.002286262 2026-02-21T01:43:15.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.004541316 2026-02-21T01:43:15.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:15.962 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2026-02-21T01:43:15.963 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-21T01:43:15.963 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:16.041 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2026-02-21T01:43:16.042 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2026-02-21T01:43:16.043 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.043 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2026-02-21T01:43:16.044 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:16.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .004541316 + 0.000315248 2026-02-21T01:43:16.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.004856564 2026-02-21T01:43:16.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:16.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2026-02-21T01:43:16.123 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-21T01:43:16.123 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:16.206 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2026-02-21T01:43:16.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2026-02-21T01:43:16.207 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.208 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2026-02-21T01:43:16.208 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:16.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .004856564 + 0.003291975 2026-02-21T01:43:16.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.008148539 2026-02-21T01:43:16.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:16.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2026-02-21T01:43:16.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-21T01:43:16.289 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:16.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2026-02-21T01:43:16.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2026-02-21T01:43:16.372 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2026-02-21T01:43:16.373 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:16.452 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .008148539 + 0.00189267 2026-02-21T01:43:16.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.010041209 2026-02-21T01:43:16.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:16.453 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2026-02-21T01:43:16.453 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-21T01:43:16.453 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:16.533 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2026-02-21T01:43:16.534 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-21T01:43:16.535 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.535 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2026-02-21T01:43:16.536 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:16.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .010041209 + 0.001868235 2026-02-21T01:43:16.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.011909444 2026-02-21T01:43:16.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:16.617 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2026-02-21T01:43:16.617 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-21T01:43:16.617 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:16.696 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2026-02-21T01:43:16.696 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2026-02-21T01:43:16.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.698 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2026-02-21T01:43:16.698 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:16.777 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .011909444 + 0.003446308 2026-02-21T01:43:16.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.015355752 2026-02-21T01:43:16.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2026-02-21T01:43:16.778 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2026-02-21T01:43:16.778 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-21T01:43:16.779 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2026-02-21T01:43:16.860 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2026-02-21T01:43:16.860 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2026-02-21T01:43:16.861 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.861 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2026-02-21T01:43:16.862 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2026-02-21T01:43:16.943 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .015355752 + 0.003695505 2026-02-21T01:43:16.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.019051257 2026-02-21T01:43:16.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2026-02-21T01:43:16.944 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '.019051257 > 0.0' 2026-02-21T01:43:16.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2026-02-21T01:43:16.945 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:16.945 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:16.947 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:16.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:16.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:16.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:17.057 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:17.057 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:17.058 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:17.058 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:17.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:17.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:17.060 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:17.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.061 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:17.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:17.062 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:17.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:17.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:43:17.102 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:17.102 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.102 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:43:17.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:17.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:17.105 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:17.105 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:17.106 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:17.106 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:17.107 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:17.107 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:17.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:17.108 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:17.109 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:17.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:17.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:17.111 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:17.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:17.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:43:17.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:17.114 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.115 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:43:17.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/test-snaptrim-stats 0 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/test-snaptrim-stats 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/test-snaptrim-stats KILL 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:17.117 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:17.118 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:17.119 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:17.119 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:17.120 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:17.121 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:17.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:17.122 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:17.122 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:17.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.123 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:17.124 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:17.124 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.125 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:17.126 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T01:43:17.126 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/test-snaptrim-stats 2026-02-21T01:43:17.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:17.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.128 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.153313 2026-02-21T01:43:17.128 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.153313 2026-02-21T01:43:17.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:17.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:17.129 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-21T01:43:17.130 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:43:17.130 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:43:17.181 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2026-02-21T01:43:17.181 DEBUG:teuthology.orchestra.run.trial106:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36d388f27052dd940bdbaf543bb05bdba0acc155 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2026-02-21T01:43:17.234 INFO:tasks.workunit.client.0.trial106.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/ver-health 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2026-02-21T01:43:17.238 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2026-02-21T01:43:17.239 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2026-02-21T01:43:17.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none ' 2026-02-21T01:43:17.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2026-02-21T01:43:17.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2026-02-21T01:43:17.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2026-02-21T01:43:17.239 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2026-02-21T01:43:17.240 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2026-02-21T01:43:17.240 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:TEST_check_version_health_2 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:TEST_check_version_health_3' 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ver-health 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ver-health 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:17.242 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:17.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:17.243 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:17.245 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:17.245 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:17.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:17.246 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:17.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:17.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:17.248 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:17.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:17.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:17.250 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:17.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:17.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:43:17.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:17.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.253 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.253 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:43:17.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:17.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:17.254 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ver-health 2026-02-21T01:43:17.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:43:17.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.256 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.166630 2026-02-21T01:43:17.257 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ver-health 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ver-health 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:43:17.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:17.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:43:17.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:17.259 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:17.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:17.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:17.259 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:17.261 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:17.262 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:17.262 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:17.263 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:17.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:17.264 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:17.264 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:17.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:17.265 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:17.266 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:17.267 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:17.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:17.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:43:17.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:17.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.269 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:43:17.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:17.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:17.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ver-health 2026-02-21T01:43:17.272 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:43:17.272 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.272 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.272 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.166630 2026-02-21T01:43:17.274 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:43:17.274 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:43:17.274 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:43:17.274 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/a 2026-02-21T01:43:17.275 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.298 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:17.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-21T01:43:17.328 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:43:17.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:43:17.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:17.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:43:17.329 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.330 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.331 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.a.asok 2026-02-21T01:43:17.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:17.331 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.a.asok config get fsid 2026-02-21T01:43:17.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:43:17.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:17.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:43:17.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.386 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.a.asok 2026-02-21T01:43:17.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:17.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.a.asok config get mon_host 2026-02-21T01:43:17.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-21T01:43:17.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:43:17.439 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:43:17.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:43:17.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:43:17.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/b 2026-02-21T01:43:17.440 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-21T01:43:17.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:43:17.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:17.458 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:17.459 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:17.459 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.459 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.460 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:17.460 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-21T01:43:17.489 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:43:17.491 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:43:17.491 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:17.491 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:43:17.491 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.492 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:43:17.493 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:17.493 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get fsid 2026-02-21T01:43:17.546 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:43:17.547 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:43:17.548 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:17.548 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.548 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:43:17.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:17.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get mon_host 2026-02-21T01:43:17.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2026-02-21T01:43:17.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:43:17.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:43:17.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:43:17.604 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/0 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:43:17.605 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:17.606 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:17.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:17.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:17.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:17.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:43:17.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:43:17.607 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/0 2026-02-21T01:43:17.608 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:43:17.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b031f900-8a2a-4a83-857b-6d4f0385e5c8 2026-02-21T01:43:17.608 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b031f900-8a2a-4a83-857b-6d4f0385e5c8' 2026-02-21T01:43:17.608 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 b031f900-8a2a-4a83-857b-6d4f0385e5c8 2026-02-21T01:43:17.609 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:43:17.619 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC1DZlpCgkRJRAAYM4X/4kZdc+12usFQGaeNA== 2026-02-21T01:43:17.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC1DZlpCgkRJRAAYM4X/4kZdc+12usFQGaeNA=="}' 2026-02-21T01:43:17.620 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b031f900-8a2a-4a83-857b-6d4f0385e5c8 -i td/ver-health/0/new.json 2026-02-21T01:43:23.740 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:23.750 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/0/new.json 2026-02-21T01:43:23.751 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1DZlpCgkRJRAAYM4X/4kZdc+12usFQGaeNA== --osd-uuid b031f900-8a2a-4a83-857b-6d4f0385e5c8 2026-02-21T01:43:23.775 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:23.775+0000 7fdb6640ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:23.777 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:23.775+0000 7fdb6640ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:23.779 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:23.779+0000 7fdb6640ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:23.779 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:23.779+0000 7fdb6640ba40 -1 bdev(0x563d33c52c00 td/ver-health/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:43:23.779 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:23.779+0000 7fdb6640ba40 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2026-02-21T01:43:24.449 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/0/keyring 2026-02-21T01:43:24.450 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:43:24.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:43:24.451 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:43:24.451 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:43:24.575 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:43:24.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:43:24.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:43:24.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:43:24.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:43:24.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:43:24.591 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:24.591+0000 7f12dede0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:24.592 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:24.591+0000 7f12dede0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:24.593 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:24.595+0000 7f12dede0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:24.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:43:24.689 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:24.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:24.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:43:24.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:24.995 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:24.995+0000 7f12dede0a40 -1 Falling back to public interface 2026-02-21T01:43:25.609 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:25.607+0000 7f12dede0a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:43:25.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:25.797 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:25.797 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:25.798 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:43:25.798 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:25.798 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:43:25.905 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:26.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:26.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:26.907 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:43:26.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:43:26.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:26.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:43:27.014 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4200665680,v1:127.0.0.1:6801/4200665680] [v2:127.0.0.1:6802/4200665680,v1:127.0.0.1:6803/4200665680] exists,up b031f900-8a2a-4a83-857b-6d4f0385e5c8 2026-02-21T01:43:27.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:43:27.017 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/1 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:27.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:27.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:43:27.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:43:27.021 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/1 2026-02-21T01:43:27.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:43:27.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5b617ce5-a937-4aec-9252-22e623611991 2026-02-21T01:43:27.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5b617ce5-a937-4aec-9252-22e623611991' 2026-02-21T01:43:27.022 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 5b617ce5-a937-4aec-9252-22e623611991 2026-02-21T01:43:27.022 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:43:27.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC/DZlpwqAKAhAAtrn+4v/bD9d80tq1+/WfvA== 2026-02-21T01:43:27.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC/DZlpwqAKAhAAtrn+4v/bD9d80tq1+/WfvA=="}' 2026-02-21T01:43:27.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5b617ce5-a937-4aec-9252-22e623611991 -i td/ver-health/1/new.json 2026-02-21T01:43:27.144 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:27.151 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/1/new.json 2026-02-21T01:43:27.152 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/DZlpwqAKAhAAtrn+4v/bD9d80tq1+/WfvA== --osd-uuid 5b617ce5-a937-4aec-9252-22e623611991 2026-02-21T01:43:27.167 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.167+0000 7f5a740faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:27.169 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.167+0000 7f5a740faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:27.170 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.171+0000 7f5a740faa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:27.170 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.171+0000 7f5a740faa40 -1 bdev(0x5609905a0c00 td/ver-health/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:43:27.170 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.171+0000 7f5a740faa40 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2026-02-21T01:43:27.820 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/1/keyring 2026-02-21T01:43:27.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:43:27.821 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:43:27.822 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:43:27.822 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:43:27.947 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:43:27.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:43:27.948 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:43:27.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:43:27.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:43:27.949 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:43:27.963 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.963+0000 7fec972b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:27.965 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.963+0000 7fec972b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:27.966 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:27.967+0000 7fec972b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:28.060 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:43:28.168 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:28.311 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:28.311+0000 7fec972b5a40 -1 Falling back to public interface 2026-02-21T01:43:28.968 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:28.967+0000 7fec972b5a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:43:29.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:29.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:29.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:43:29.169 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:29.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:29.169 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:43:29.277 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:30.278 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:30.279 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:30.279 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:43:30.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:43:30.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:30.280 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:43:30.393 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1086083010,v1:127.0.0.1:6809/1086083010] [v2:127.0.0.1:6810/1086083010,v1:127.0.0.1:6811/1086083010] exists,up 5b617ce5-a937-4aec-9252-22e623611991 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/2 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:30.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:30.395 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:43:30.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/2 2026-02-21T01:43:30.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:43:30.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=df41205e-cef3-44f9-aa15-7888b2c34437 2026-02-21T01:43:30.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 df41205e-cef3-44f9-aa15-7888b2c34437' 2026-02-21T01:43:30.399 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 df41205e-cef3-44f9-aa15-7888b2c34437 2026-02-21T01:43:30.399 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:43:30.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDCDZlpewarGBAA3XrQvnsiupaSBibVrCAoNw== 2026-02-21T01:43:30.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDCDZlpewarGBAA3XrQvnsiupaSBibVrCAoNw=="}' 2026-02-21T01:43:30.411 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new df41205e-cef3-44f9-aa15-7888b2c34437 -i td/ver-health/2/new.json 2026-02-21T01:43:30.532 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:43:30.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/2/new.json 2026-02-21T01:43:30.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCDZlpewarGBAA3XrQvnsiupaSBibVrCAoNw== --osd-uuid df41205e-cef3-44f9-aa15-7888b2c34437 2026-02-21T01:43:30.556 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:30.555+0000 7fe6fb207a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:30.558 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:30.559+0000 7fe6fb207a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:30.559 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:30.559+0000 7fe6fb207a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:30.559 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:30.559+0000 7fe6fb207a40 -1 bdev(0x562ffab86c00 td/ver-health/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:43:30.559 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:30.559+0000 7fe6fb207a40 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2026-02-21T01:43:31.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/2/keyring 2026-02-21T01:43:31.188 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:43:31.189 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:43:31.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:43:31.189 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:43:31.317 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:43:31.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:43:31.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:43:31.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:43:31.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:43:31.318 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:43:31.332 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:31.331+0000 7febe7ca6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:31.334 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:31.335+0000 7febe7ca6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:31.335 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:31.335+0000 7febe7ca6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:31.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:43:31.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:43:31.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:43:31.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:43:31.429 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:43:31.430 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:31.430 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:31.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:43:31.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:31.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:43:31.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:31.659 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:31.659+0000 7febe7ca6a40 -1 Falling back to public interface 2026-02-21T01:43:32.263 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:32.263+0000 7febe7ca6a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:43:32.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:32.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:32.543 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:32.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:43:32.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:32.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:43:32.656 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:33.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:33.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:33.657 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:43:33.657 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:43:33.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:33.658 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:43:33.772 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2735585437,v1:127.0.0.1:6817/2735585437] [v2:127.0.0.1:6818/2735585437,v1:127.0.0.1:6819/2735585437] exists,up df41205e-cef3-44f9-aa15-7888b2c34437 2026-02-21T01:43:33.772 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:43:33.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:43:33.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:43:33.773 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2026-02-21T01:43:38.774 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:38.879 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_OK 2026-02-21T01:43:38.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:38.889 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2026-02-21T01:43:38.998 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2026-02-21T01:43:38.999 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:38.999 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:38.999 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:39.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:39.000 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:39.110 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/1 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:39.111 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:39.112 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:43:39.113 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/1 2026-02-21T01:43:39.114 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/1/whoami 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:43:39.115 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:43:39.126 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:39.127+0000 7f12c7fc6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:39.128 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:39.127+0000 7f12c7fc6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:39.128 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:39.127+0000 7f12c7fc6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:39.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-21T01:43:39.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:43:39.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:39.222 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:43:39.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:39.515 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:39.515+0000 7f12c7fc6a40 -1 Falling back to public interface 2026-02-21T01:43:40.094 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:40.090+0000 7f12c7fc6a40 -1 osd.1 13 log_to_monitors true 2026-02-21T01:43:40.333 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:40.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:40.334 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:40.334 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:43:40.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:40.335 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:43:40.446 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 14 last_clean_interval [9,13) [v2:127.0.0.1:6808/1746416276,v1:127.0.0.1:6809/1746416276] [v2:127.0.0.1:6810/1746416276,v1:127.0.0.1:6811/1746416276] exists,up 5b617ce5-a937-4aec-9252-22e623611991 2026-02-21T01:43:40.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:43:40.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:43:40.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:40.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:40.559 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:41.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:41.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:41.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:41.560 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:41.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:42.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:42.674 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:42.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:42.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:42.789 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:43.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:43.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:43.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:43.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:43.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:44.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:44.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:44.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:44.908 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:45.023 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:46.024 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:46.024 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:46.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:46.025 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:46.137 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:47.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:47.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:47.138 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:47.139 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:47.249 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:48.251 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:48.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:48.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:48.252 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:48.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-21T01:43:48.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-21T01:43:48.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-21T01:43:48.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-21T01:43:48.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-21T01:43:48.366 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:48.472 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2026-02-21T01:43:48.472 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2026-02-21T01:43:48.472 INFO:tasks.workunit.client.0.trial106.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2026-02-21T01:43:48.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:48.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:43:48.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:48.600 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2026-02-21T01:43:48.713 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:48.714 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2026-02-21T01:43:48.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2026-02-21T01:43:48.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:48.842 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:48.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:48.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:48.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:48.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:48.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:43:48.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2026-02-21T01:43:48.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:43:48.951 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/2 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-21T01:43:48.952 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:48.953 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:43:48.954 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/2 2026-02-21T01:43:48.955 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/2/whoami 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:43:48.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:43:48.970 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:48.970+0000 7f5f119efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:48.972 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:48.970+0000 7f5f119efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:48.973 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:48.970+0000 7f5f119efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:49.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-21T01:43:49.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:43:49.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:43:49.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:43:49.062 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:43:49.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:49.063 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:49.063 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:43:49.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:49.064 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:43:49.172 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:49.367 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:49.366+0000 7f5f119efa40 -1 Falling back to public interface 2026-02-21T01:43:50.032 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:50.030+0000 7f5f119efa40 -1 osd.2 17 log_to_monitors true 2026-02-21T01:43:50.173 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:50.174 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:50.174 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:43:50.174 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:50.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:50.175 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:43:50.287 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [13,17) [v2:127.0.0.1:6816/3508242083,v1:127.0.0.1:6817/3508242083] [v2:127.0.0.1:6818/3508242083,v1:127.0.0.1:6819/3508242083] exists,up df41205e-cef3-44f9-aa15-7888b2c34437 2026-02-21T01:43:50.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:43:50.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:50.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:43:50.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/0 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:43:50.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:50.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:50.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:43:50.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:43:50.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/0 2026-02-21T01:43:50.402 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:43:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:43:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:43:50.403 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/0/whoami 2026-02-21T01:43:50.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:43:50.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:43:50.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:43:50.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:43:50.417 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:50.414+0000 7face08bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:50.419 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:50.418+0000 7face08bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:50.420 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:50.418+0000 7face08bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:43:50.514 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:43:50.515 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:43:50.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:50.516 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:43:50.622 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:43:50.796 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:50.794+0000 7face08bda40 -1 Falling back to public interface 2026-02-21T01:43:51.426 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:43:51.426+0000 7face08bda40 -1 osd.0 20 log_to_monitors true 2026-02-21T01:43:51.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:43:51.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:43:51.624 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:43:51.625 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:43:51.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:43:51.625 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:43:51.729 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6800/533232551,v1:127.0.0.1:6801/533232551] [v2:127.0.0.1:6802/533232551,v1:127.0.0.1:6803/533232551] exists,up b031f900-8a2a-4a83-857b-6d4f0385e5c8 2026-02-21T01:43:51.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:43:51.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:43:51.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:43:51.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:51.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:43:51.839 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:43:52.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:43:52.840 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:43:52.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:43:52.841 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:43:52.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-21T01:43:52.955 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-21T01:43:52.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-21T01:43:52.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-21T01:43:52.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-21T01:43:52.956 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:53.061 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2026-02-21T01:43:53.062 INFO:tasks.workunit.client.0.trial106.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2026-02-21T01:43:53.062 INFO:tasks.workunit.client.0.trial106.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2026-02-21T01:43:53.062 INFO:tasks.workunit.client.0.trial106.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2026-02-21T01:43:53.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:53.070 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:43:53.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:53.198 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2026-02-21T01:43:53.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:53.311 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2026-02-21T01:43:53.425 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2026-02-21T01:43:53.426 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2026-02-21T01:43:53.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2026-02-21T01:43:53.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:43:53.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:53.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:43:53.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:53.538 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:53.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:53.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:53.539 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:53.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:53.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:53.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:53.650 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:53.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:53.651 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:53.651 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:53.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:53.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:53.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:53.653 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:53.654 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:53.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:53.655 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:43:53.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:53.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.668 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.668 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ver-health 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ver-health 2026-02-21T01:43:53.670 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:53.671 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:53.672 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:53.673 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:53.673 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:53.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:53.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:53.675 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:53.676 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:53.676 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:53.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:53.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:53.677 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:53.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:53.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:53.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:43:53.680 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:53.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.681 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.681 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:43:53.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:53.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:53.682 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ver-health 2026-02-21T01:43:53.683 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:43:53.683 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.684 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.684 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.166630 2026-02-21T01:43:53.685 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:43:53.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:43:53.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:43:53.687 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ver-health 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ver-health 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:43:53.688 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:43:53.689 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:43:53.689 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:43:53.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:43:53.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:43:53.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:43:53.689 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:43:53.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:43:53.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:43:53.690 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:43:53.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:43:53.692 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:43:53.692 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:43:53.693 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:53.693 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:43:53.694 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:43:53.694 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:43:53.695 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:43:53.696 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:43:53.696 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:43:53.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:43:53.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.697 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:43:53.698 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:43:53.699 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:43:53.699 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ver-health 2026-02-21T01:43:53.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:43:53.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.700 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.700 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.166630 2026-02-21T01:43:53.702 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:43:53.703 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:43:53.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:43:53.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/a 2026-02-21T01:43:53.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-21T01:43:53.725 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:43:53.725 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:53.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:53.726 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:53.726 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.726 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.726 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:53.726 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2026-02-21T01:43:53.753 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:43:53.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:43:53.753 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:53.754 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:43:53.754 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:43:53.754 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:43:53.754 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:43:53.754 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:43:53.754 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:43:53.755 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:53.755 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.755 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.755 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.a.asok 2026-02-21T01:43:53.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:53.755 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.a.asok config get fsid 2026-02-21T01:43:53.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:43:53.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:53.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:43:53.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:43:53.811 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:43:53.811 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:43:53.812 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:43:53.812 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:43:53.813 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:53.813 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.813 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.813 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.a.asok 2026-02-21T01:43:53.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:53.813 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.a.asok config get mon_host 2026-02-21T01:43:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-21T01:43:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:43:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:43:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:43:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:43:53.866 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/b 2026-02-21T01:43:53.867 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-21T01:43:53.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:43:53.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:53.885 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:53.886 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:53.886 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.886 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.886 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:53.886 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2026-02-21T01:43:53.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:43:53.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:43:53.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:53.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:43:53.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:43:53.913 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.914 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:43:53.915 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:53.915 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get fsid 2026-02-21T01:43:53.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:43:53.971 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:43:53.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:43:53.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:43:53.972 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:43:53.972 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:43:53.972 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:43:53.972 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:43:53.973 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:43:53.973 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:53.973 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:53.973 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:43:53.973 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:43:53.974 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get mon_host 2026-02-21T01:43:54.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2026-02-21T01:43:54.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:43:54.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:43:54.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:43:54.029 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/0 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:43:54.030 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:43:54.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/0 2026-02-21T01:43:54.032 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:43:54.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ee1da396-6a46-4534-9ecb-20dab70dddac 2026-02-21T01:43:54.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ee1da396-6a46-4534-9ecb-20dab70dddac' 2026-02-21T01:43:54.034 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 ee1da396-6a46-4534-9ecb-20dab70dddac 2026-02-21T01:43:54.034 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:43:54.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDaDZlpyoTyAhAALwhlzzGcjXQ61OOfdpWxTA== 2026-02-21T01:43:54.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDaDZlpyoTyAhAALwhlzzGcjXQ61OOfdpWxTA=="}' 2026-02-21T01:43:54.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ee1da396-6a46-4534-9ecb-20dab70dddac -i td/ver-health/0/new.json 2026-02-21T01:44:00.166 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:00.176 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/0/new.json 2026-02-21T01:44:00.177 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDaDZlpyoTyAhAALwhlzzGcjXQ61OOfdpWxTA== --osd-uuid ee1da396-6a46-4534-9ecb-20dab70dddac 2026-02-21T01:44:00.193 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:00.190+0000 7fbd9d98ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:00.195 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:00.194+0000 7fbd9d98ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:00.196 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:00.194+0000 7fbd9d98ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:00.197 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:00.194+0000 7fbd9d98ea40 -1 bdev(0x560190c76c00 td/ver-health/0/block) open stat got: (1) Operation not permitted 2026-02-21T01:44:00.197 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:00.194+0000 7fbd9d98ea40 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2026-02-21T01:44:00.969 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/0/keyring 2026-02-21T01:44:00.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:44:00.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:44:00.971 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:44:00.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:44:01.089 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:44:01.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:44:01.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:01.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:44:01.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:44:01.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:44:01.114 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:01.109+0000 7f11d850fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:01.116 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:01.113+0000 7f11d850fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:01.117 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:01.113+0000 7f11d850fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:01.199 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:01.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:01.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:01.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:01.312 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:01.536 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:01.533+0000 7f11d850fa40 -1 Falling back to public interface 2026-02-21T01:44:02.160 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:02.157+0000 7f11d850fa40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:44:02.314 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:02.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:02.315 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:02.315 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:02.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:02.316 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:02.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:03.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:03.433 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:03.433 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:44:03.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:44:03.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:03.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:03.547 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1408513483,v1:127.0.0.1:6801/1408513483] [v2:127.0.0.1:6802/1408513483,v1:127.0.0.1:6803/1408513483] exists,up ee1da396-6a46-4534-9ecb-20dab70dddac 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/1 2026-02-21T01:44:03.548 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:03.549 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:44:03.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/1 2026-02-21T01:44:03.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:44:03.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d021462a-9e6a-43f0-988a-2231582fec3e 2026-02-21T01:44:03.553 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d021462a-9e6a-43f0-988a-2231582fec3e' 2026-02-21T01:44:03.553 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 d021462a-9e6a-43f0-988a-2231582fec3e 2026-02-21T01:44:03.553 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:44:03.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjDZlpWNDFIRAA9Kj0B7ZJTtBsD+CB0gfqfg== 2026-02-21T01:44:03.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjDZlpWNDFIRAA9Kj0B7ZJTtBsD+CB0gfqfg=="}' 2026-02-21T01:44:03.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d021462a-9e6a-43f0-988a-2231582fec3e -i td/ver-health/1/new.json 2026-02-21T01:44:03.686 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:03.696 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/1/new.json 2026-02-21T01:44:03.697 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjDZlpWNDFIRAA9Kj0B7ZJTtBsD+CB0gfqfg== --osd-uuid d021462a-9e6a-43f0-988a-2231582fec3e 2026-02-21T01:44:03.720 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:03.717+0000 7f9d46b24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:03.721 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:03.717+0000 7f9d46b24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:03.723 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:03.721+0000 7f9d46b24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:03.723 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:03.721+0000 7f9d46b24a40 -1 bdev(0x562241be0c00 td/ver-health/1/block) open stat got: (1) Operation not permitted 2026-02-21T01:44:03.723 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:03.721+0000 7f9d46b24a40 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2026-02-21T01:44:04.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/1/keyring 2026-02-21T01:44:04.330 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:44:04.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:44:04.331 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:44:04.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:44:04.457 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:44:04.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:44:04.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:04.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:44:04.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:44:04.457 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:44:04.473 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:04.469+0000 7f8f67410a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:04.474 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:04.473+0000 7f8f67410a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:04.475 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:04.473+0000 7f8f67410a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:04.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:44:04.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:04.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:44:04.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:04.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:04.570 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:04.571 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:04.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:04.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:04.571 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:04.677 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:04.808 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:04.805+0000 7f8f67410a40 -1 Falling back to public interface 2026-02-21T01:44:05.373 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:05.369+0000 7f8f67410a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:44:05.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:05.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:05.678 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:05.678 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:05.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:05.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:05.792 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:06.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:06.795 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:06.795 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:44:06.796 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:44:06.796 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:06.796 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:06.905 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/281572024,v1:127.0.0.1:6809/281572024] [v2:127.0.0.1:6810/281572024,v1:127.0.0.1:6811/281572024] exists,up d021462a-9e6a-43f0-988a-2231582fec3e 2026-02-21T01:44:06.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:06.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:06.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:06.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2026-02-21T01:44:06.909 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/2 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-21T01:44:06.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:06.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/2 2026-02-21T01:44:06.912 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:44:06.913 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ee85d0e1-cbba-4726-914f-7d12ebabbcdf 2026-02-21T01:44:06.913 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 ee85d0e1-cbba-4726-914f-7d12ebabbcdf 2026-02-21T01:44:06.914 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ee85d0e1-cbba-4726-914f-7d12ebabbcdf' 2026-02-21T01:44:06.914 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:44:06.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDmDZlpk54FNxAAvcalw/r+DdVwL0VqPpKc7Q== 2026-02-21T01:44:06.921 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDmDZlpk54FNxAAvcalw/r+DdVwL0VqPpKc7Q=="}' 2026-02-21T01:44:06.922 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ee85d0e1-cbba-4726-914f-7d12ebabbcdf -i td/ver-health/2/new.json 2026-02-21T01:44:07.044 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:44:07.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/2/new.json 2026-02-21T01:44:07.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmDZlpk54FNxAAvcalw/r+DdVwL0VqPpKc7Q== --osd-uuid ee85d0e1-cbba-4726-914f-7d12ebabbcdf 2026-02-21T01:44:07.068 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.065+0000 7fb31744fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:07.070 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.065+0000 7fb31744fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:07.070 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.069+0000 7fb31744fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:07.071 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.069+0000 7fb31744fa40 -1 bdev(0x55e31ce24c00 td/ver-health/2/block) open stat got: (1) Operation not permitted 2026-02-21T01:44:07.071 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.069+0000 7fb31744fa40 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2026-02-21T01:44:07.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/2/keyring 2026-02-21T01:44:07.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:44:07.706 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:44:07.706 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:44:07.707 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:44:07.825 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:44:07.826 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:44:07.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:07.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:44:07.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:44:07.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:44:07.850 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.845+0000 7fd7fb70ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:07.852 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.849+0000 7fd7fb70ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:07.853 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:07.849+0000 7fd7fb70ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:07.936 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:07.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:07.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:07.937 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:08.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:08.192 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:08.189+0000 7fd7fb70ca40 -1 Falling back to public interface 2026-02-21T01:44:08.765 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:08.761+0000 7fd7fb70ca40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:44:09.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:09.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:09.047 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:09.048 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:09.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:09.048 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:09.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:10.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:10.156 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:10.157 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:44:10.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-21T01:44:10.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:10.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:10.265 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2359728957,v1:127.0.0.1:6817/2359728957] [v2:127.0.0.1:6818/2359728957,v1:127.0.0.1:6819/2359728957] exists,up ee85d0e1-cbba-4726-914f-7d12ebabbcdf 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ver-health 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ver-health/x 2026-02-21T01:44:10.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:44:10.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:44:10.375 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:10.375 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:10.375 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:10.375 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:10.375 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:10.375 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:10.376 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:44:10.376 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:44:10.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2026-02-21T01:44:10.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ver-health 2026-02-21T01:44:10.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T01:44:10.397 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=y 2026-02-21T01:44:10.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T01:44:10.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ver-health/y 2026-02-21T01:44:10.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T01:44:10.508 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T01:44:10.508 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:10.508 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:10.508 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:10.509 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:10.509 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:10.509 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:10.509 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:44:10.510 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ver-health 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=m 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ver-health/m 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:10.528 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:10.529 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:10.529 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:10.529 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:10.529 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:10.529 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:10.546 INFO:tasks.workunit.client.0.trial106.stdout:starting mds.m at 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ver-health 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=n 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ver-health/n 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:10.550 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:10.551 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:10.551 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:10.551 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:10.551 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:10.551 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:10.570 INFO:tasks.workunit.client.0.trial106.stdout:starting mds.n at 2026-02-21T01:44:10.574 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2026-02-21T01:44:15.576 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:15.851 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_OK 2026-02-21T01:44:15.861 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:15.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2026-02-21T01:44:16.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2026-02-21T01:44:16.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:16.160 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:16.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:16.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:16.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/b 2026-02-21T01:44:16.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2026-02-21T01:44:16.281 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:16.277+0000 7f3dde57fec0 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2026-02-21T01:44:16.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:44:16.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:16.284 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:16.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:16.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:16.284 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:16.285 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:16.285 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2026-02-21T01:44:16.319 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:44:16.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:44:16.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:44:16.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:44:16.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:44:16.320 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:44:16.321 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:44:16.322 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get fsid 2026-02-21T01:44:16.376 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:44:16.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:44:16.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:44:16.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:44:16.377 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:44:16.378 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:44:16.378 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:44:16.378 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:44:16.378 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:44:16.379 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:16.379 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:16.379 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:44:16.379 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:44:16.379 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get mon_host 2026-02-21T01:44:16.431 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2026-02-21T01:44:16.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:16.431 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:16.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:16.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:16.432 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:16.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:16.540 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ver-health 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=m 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ver-health/m 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:16.541 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:16.542 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:16.542 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:16.563 INFO:tasks.workunit.client.0.trial106.stdout:starting mds.m at 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-21T01:44:16.568 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:16.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:16.569 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-21T01:44:16.827 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:44:17.828 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:44:17.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:17.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:17.829 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-21T01:44:18.086 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:44:19.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:44:19.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:19.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:19.088 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-21T01:44:19.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-21T01:44:19.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-21T01:44:19.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-21T01:44:19.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-21T01:44:19.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-21T01:44:19.347 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:19.638 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_WARN There are daemons running an older version of ceph 2026-02-21T01:44:19.638 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2026-02-21T01:44:19.638 INFO:tasks.workunit.client.0.trial106.stdout: mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test 2026-02-21T01:44:19.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:19.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2026-02-21T01:44:19.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:19.945 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2026-02-21T01:44:20.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:20.247 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2026-02-21T01:44:20.536 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:153: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.2 2026-02-21T01:44:20.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:20.537 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:20.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:20.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:20.538 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:20.647 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: activate_osd td/ver-health 2 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/2 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:20.648 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:20.649 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:44:20.650 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/2 2026-02-21T01:44:20.651 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:44:20.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-21T01:44:20.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:20.652 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/2/whoami 2026-02-21T01:44:20.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-21T01:44:20.652 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:44:20.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:44:20.653 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:44:20.672 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:20.669+0000 7feea193aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:20.673 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:20.669+0000 7feea193aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:20.674 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:20.669+0000 7feea193aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:20.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-21T01:44:20.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:20.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:44:20.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:20.863 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:20.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:20.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:20.864 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:20.864 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:20.865 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:21.036 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:21.033+0000 7feea193aa40 -1 Falling back to public interface 2026-02-21T01:44:21.078 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:21.654 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:21.653+0000 7feea193aa40 -1 osd.2 14 log_to_monitors true 2026-02-21T01:44:22.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:22.079 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:22.080 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:22.080 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:22.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:22.081 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:22.287 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 17 up_thru 0 down_at 15 last_clean_interval [13,14) [v2:127.0.0.1:6816/2558637133,v1:127.0.0.1:6817/2558637133] [v2:127.0.0.1:6818/2558637133,v1:127.0.0.1:6819/2558637133] exists,up ee85d0e1-cbba-4726-914f-7d12ebabbcdf 2026-02-21T01:44:22.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:22.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:22.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:22.288 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:155: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.0 2026-02-21T01:44:22.288 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:22.289 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:22.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:22.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:22.289 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: ceph_debug_version_for_testing=02.00.00-gversion-test 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: activate_osd td/ver-health 0 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/0 2026-02-21T01:44:22.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:22.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:22.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:22.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:44:22.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:44:22.402 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/0 2026-02-21T01:44:22.402 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:44:22.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:44:22.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:22.403 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/0/whoami 2026-02-21T01:44:22.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:44:22.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:44:22.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:44:22.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:44:22.418 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:22.420+0000 7f34c2c18a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:22.420 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:22.420+0000 7f34c2c18a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:22.421 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:22.420+0000 7f34c2c18a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:22.614 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:22.615 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:22.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:22.615 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:22.835 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:22.844 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:22.844+0000 7f34c2c18a40 -1 Falling back to public interface 2026-02-21T01:44:23.484 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:23.484+0000 7f34c2c18a40 -1 osd.0 15 log_to_monitors true 2026-02-21T01:44:23.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:23.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:23.836 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:23.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:23.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:23.836 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:24.051 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 21 up_thru 0 down_at 18 last_clean_interval [5,15) [v2:127.0.0.1:6800/1009075234,v1:127.0.0.1:6801/1009075234] [v2:127.0.0.1:6802/1009075234,v1:127.0.0.1:6803/1009075234] exists,up ee1da396-6a46-4534-9ecb-20dab70dddac 2026-02-21T01:44:24.051 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:158: TEST_check_version_health_2: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:24.052 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:44:24.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-21T01:44:24.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-21T01:44:24.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-21T01:44:24.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-21T01:44:24.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-21T01:44:24.299 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:160: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:24.582 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2026-02-21T01:44:24.582 INFO:tasks.workunit.client.0.trial106.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2026-02-21T01:44:24.582 INFO:tasks.workunit.client.0.trial106.stdout: mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test 2026-02-21T01:44:24.583 INFO:tasks.workunit.client.0.trial106.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2026-02-21T01:44:24.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: ceph health 2026-02-21T01:44:24.593 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:44:24.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:24.862 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:44:25.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:25.159 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2026-02-21T01:44:25.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:25.447 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: grep -q 'mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test' 2026-02-21T01:44:25.742 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: ceph health detail 2026-02-21T01:44:25.743 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2026-02-21T01:44:26.031 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:26.032 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:26.344 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:26.345 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:44:26.346 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:44:26.347 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:44:26.349 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:44:26.349 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:44:26.349 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:44:26.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:44:26.350 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:44:26.351 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:44:26.351 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:44:26.352 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:44:26.353 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:44:26.353 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:44:26.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:44:26.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.368 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.368 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:44:26.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ver-health 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ver-health 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:26.371 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:26.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:26.373 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:26.373 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:44:26.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:44:26.374 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:44:26.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:44:26.375 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:44:26.376 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:44:26.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:44:26.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:44:26.377 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:44:26.378 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:44:26.378 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:44:26.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:44:26.380 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:44:26.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:44:26.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.381 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.381 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:44:26.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:44:26.382 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:44:26.383 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ver-health 2026-02-21T01:44:26.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:44:26.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.384 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.384 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.166630 2026-02-21T01:44:26.385 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:44:26.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_3 td/ver-health 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:170: TEST_check_version_health_3: local dir=td/ver-health 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:173: TEST_check_version_health_3: CEPH_ARGS='--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:175: TEST_check_version_health_3: setup td/ver-health 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ver-health 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ver-health 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:44:26.388 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:26.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:26.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:26.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:26.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:26.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:26.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:44:26.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:44:26.391 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:44:26.391 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:44:26.392 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:44:26.392 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:44:26.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:44:26.393 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:44:26.394 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:44:26.394 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:44:26.395 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:44:26.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:44:26.396 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:44:26.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:44:26.397 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.398 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:44:26.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:44:26.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:44:26.399 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ver-health 2026-02-21T01:44:26.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T01:44:26.400 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.401 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.401 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.166630 2026-02-21T01:44:26.402 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T01:44:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T01:44:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T01:44:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2026-02-21T01:44:26.403 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:178: TEST_check_version_health_3: run_mon td/ver-health a --public-addr=127.0.0.1:7165 2026-02-21T01:44:26.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:44:26.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:44:26.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T01:44:26.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:44:26.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/a 2026-02-21T01:44:26.404 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 2026-02-21T01:44:26.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:44:26.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:26.426 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:26.426 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:26.426 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.426 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.427 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:26.427 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 2026-02-21T01:44:26.454 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:44:26.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T01:44:26.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:44:26.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:44:26.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:44:26.455 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.a.asok 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:44:26.457 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.a.asok config get fsid 2026-02-21T01:44:26.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T01:44:26.511 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:44:26.512 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T01:44:26.512 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:44:26.512 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:44:26.513 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T01:44:26.513 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T01:44:26.513 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T01:44:26.513 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:44:26.514 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.514 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.514 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.a.asok 2026-02-21T01:44:26.514 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:44:26.514 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.a.asok config get mon_host 2026-02-21T01:44:26.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:179: TEST_check_version_health_3: run_mon td/ver-health b --public-addr=127.0.0.1:7166 2026-02-21T01:44:26.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ver-health 2026-02-21T01:44:26.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T01:44:26.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T01:44:26.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T01:44:26.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ver-health/b 2026-02-21T01:44:26.567 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 2026-02-21T01:44:26.585 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T01:44:26.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:26.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:26.586 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:26.586 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.586 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.586 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:26.587 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 2026-02-21T01:44:26.611 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T01:44:26.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T01:44:26.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:44:26.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:44:26.612 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T01:44:26.613 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:44:26.614 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:44:26.615 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get fsid 2026-02-21T01:44:26.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T01:44:26.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T01:44:26.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T01:44:26.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T01:44:26.672 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.166630/ceph-mon.b.asok 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T01:44:26.674 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.166630/ceph-mon.b.asok config get mon_host 2026-02-21T01:44:26.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:181: TEST_check_version_health_3: local start_osd_time=69 2026-02-21T01:44:26.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-21T01:44:26.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: run_osd td/ver-health 0 2026-02-21T01:44:26.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:44:26.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:44:26.731 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/0 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:26.732 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:26.733 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:26.733 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:26.733 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:26.734 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/0 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1227070c-1332-4d12-b205-e20c86230d21 2026-02-21T01:44:26.735 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1227070c-1332-4d12-b205-e20c86230d21' 2026-02-21T01:44:26.736 INFO:tasks.workunit.client.0.trial106.stdout:add osd0 1227070c-1332-4d12-b205-e20c86230d21 2026-02-21T01:44:26.736 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:44:26.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6DZlpEieFLBAAYsgVJHCQCWEo5A5OahiIvQ== 2026-02-21T01:44:26.745 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6DZlpEieFLBAAYsgVJHCQCWEo5A5OahiIvQ=="}' 2026-02-21T01:44:26.746 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1227070c-1332-4d12-b205-e20c86230d21 -i td/ver-health/0/new.json 2026-02-21T01:44:32.869 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:32.879 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/0/new.json 2026-02-21T01:44:32.880 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6DZlpEieFLBAAYsgVJHCQCWEo5A5OahiIvQ== --osd-uuid 1227070c-1332-4d12-b205-e20c86230d21 2026-02-21T01:44:32.905 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:32.904+0000 7fdaef197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:32.907 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:32.908+0000 7fdaef197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:32.908 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:32.908+0000 7fdaef197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:32.908 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:32.908+0000 7fdaef197a40 -1 memstore(td/ver-health/0) td/ver-health/0 2026-02-21T01:44:32.918 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/0/keyring 2026-02-21T01:44:32.918 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:44:32.919 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T01:44:32.919 INFO:tasks.workunit.client.0.trial106.stdout:adding osd0 key to auth repository 2026-02-21T01:44:32.920 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:44:33.044 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:44:33.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T01:44:33.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:33.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:44:33.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:44:33.045 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:44:33.061 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:33.060+0000 7fc1cff6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:33.062 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:33.060+0000 7fc1cff6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:33.063 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:33.064+0000 7fc1cff6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:33.065 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:33.064+0000 7fc1cff6ca40 -1 Falling back to public interface 2026-02-21T01:44:33.070 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:33.068+0000 7fc1cff6ca40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:33.154 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:33.266 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:34.268 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:34.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:34.269 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:34.269 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:34.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:34.270 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:44:34.382 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/765467518,v1:127.0.0.1:6801/765467518] [v2:127.0.0.1:6802/765467518,v1:127.0.0.1:6803/765467518] exists,up 1227070c-1332-4d12-b205-e20c86230d21 2026-02-21T01:44:34.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:34.385 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: run_osd td/ver-health 1 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/1 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:34.386 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:34.387 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:44:34.388 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:44:34.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/1 2026-02-21T01:44:34.389 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:44:34.389 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=171a9f42-8fa5-4002-8c09-11e5e3d63c3e 2026-02-21T01:44:34.389 INFO:tasks.workunit.client.0.trial106.stdout:add osd1 171a9f42-8fa5-4002-8c09-11e5e3d63c3e 2026-02-21T01:44:34.390 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 171a9f42-8fa5-4002-8c09-11e5e3d63c3e' 2026-02-21T01:44:34.390 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:44:34.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQACDplp12/IFxAAuEvN6UbP2MdS9dZ4M7pOGw== 2026-02-21T01:44:34.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQACDplp12/IFxAAuEvN6UbP2MdS9dZ4M7pOGw=="}' 2026-02-21T01:44:34.398 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 171a9f42-8fa5-4002-8c09-11e5e3d63c3e -i td/ver-health/1/new.json 2026-02-21T01:44:34.519 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:34.530 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/1/new.json 2026-02-21T01:44:34.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACDplp12/IFxAAuEvN6UbP2MdS9dZ4M7pOGw== --osd-uuid 171a9f42-8fa5-4002-8c09-11e5e3d63c3e 2026-02-21T01:44:34.554 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.552+0000 7f9ba14eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:34.555 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.556+0000 7f9ba14eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:34.556 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.556+0000 7f9ba14eba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:34.557 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.556+0000 7f9ba14eba40 -1 memstore(td/ver-health/1) td/ver-health/1 2026-02-21T01:44:34.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/1/keyring 2026-02-21T01:44:34.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:44:34.566 INFO:tasks.workunit.client.0.trial106.stdout:adding osd1 key to auth repository 2026-02-21T01:44:34.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T01:44:34.566 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:44:34.689 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:44:34.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T01:44:34.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:34.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:44:34.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:44:34.690 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:44:34.705 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.704+0000 7f0357298a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:34.707 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.708+0000 7f0357298a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:34.708 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.708+0000 7f0357298a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:34.710 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.708+0000 7f0357298a40 -1 Falling back to public interface 2026-02-21T01:44:34.715 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:34.716+0000 7f0357298a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:44:34.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T01:44:34.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:34.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:44:34.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:34.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:34.799 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:34.800 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:34.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:34.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:34.800 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:34.910 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:35.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:35.911 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:35.912 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:35.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:35.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:35.912 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/3848562750,v1:127.0.0.1:6809/3848562750] [v2:127.0.0.1:6810/3848562750,v1:127.0.0.1:6811/3848562750] exists,up 171a9f42-8fa5-4002-8c09-11e5e3d63c3e 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: run_osd td/ver-health 2 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ver-health 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T01:44:36.018 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ver-health/2 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:36.019 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T01:44:36.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ver-health/2 2026-02-21T01:44:36.021 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T01:44:36.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fcc14b59-be2c-4ff2-b558-a349ec1d014a 2026-02-21T01:44:36.022 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fcc14b59-be2c-4ff2-b558-a349ec1d014a' 2026-02-21T01:44:36.022 INFO:tasks.workunit.client.0.trial106.stdout:add osd2 fcc14b59-be2c-4ff2-b558-a349ec1d014a 2026-02-21T01:44:36.022 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T01:44:36.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEDplp780jAhAAu+oj8MdhDwJEnSLOtImHJw== 2026-02-21T01:44:36.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEDplp780jAhAAu+oj8MdhDwJEnSLOtImHJw=="}' 2026-02-21T01:44:36.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fcc14b59-be2c-4ff2-b558-a349ec1d014a -i td/ver-health/2/new.json 2026-02-21T01:44:36.151 INFO:tasks.workunit.client.0.trial106.stdout:2 2026-02-21T01:44:36.158 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ver-health/2/new.json 2026-02-21T01:44:36.160 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEDplp780jAhAAu+oj8MdhDwJEnSLOtImHJw== --osd-uuid fcc14b59-be2c-4ff2-b558-a349ec1d014a 2026-02-21T01:44:36.183 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.184+0000 7f12a071aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:36.185 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.184+0000 7f12a071aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:36.186 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.184+0000 7f12a071aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:36.187 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.188+0000 7f12a071aa40 -1 memstore(td/ver-health/2) td/ver-health/2 2026-02-21T01:44:36.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ver-health/2/keyring 2026-02-21T01:44:36.195 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T01:44:36.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T01:44:36.196 INFO:tasks.workunit.client.0.trial106.stdout:adding osd2 key to auth repository 2026-02-21T01:44:36.196 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T01:44:36.322 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:44:36.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T01:44:36.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:36.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T01:44:36.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T01:44:36.323 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T01:44:36.337 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.336+0000 7f5b99798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:36.339 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.340+0000 7f5b99798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:36.340 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.340+0000 7f5b99798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:36.342 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.340+0000 7f5b99798a40 -1 Falling back to public interface 2026-02-21T01:44:36.349 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:36.348+0000 7f5b99798a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:44:36.434 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:36.435 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:36.543 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:37.544 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:37.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:37.545 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:37.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:37.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:37.545 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:44:37.659 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/698690373,v1:127.0.0.1:6817/698690373] [v2:127.0.0.1:6818/698690373,v1:127.0.0.1:6819/698690373] exists,up fcc14b59-be2c-4ff2-b558-a349ec1d014a 2026-02-21T01:44:37.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:37.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:37.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:37.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:187: TEST_check_version_health_3: local warn_older_version_delay=31 2026-02-21T01:44:37.660 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:189: TEST_check_version_health_3: sleep 5 2026-02-21T01:44:42.661 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:190: TEST_check_version_health_3: ceph health detail 2026-02-21T01:44:42.761 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_OK 2026-02-21T01:44:42.769 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: ceph health detail 2026-02-21T01:44:42.770 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2026-02-21T01:44:42.893 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:193: TEST_check_version_health_3: ceph tell 'mon.*' injectargs '--mon_warn_older_version_delay 31' 2026-02-21T01:44:42.955 INFO:tasks.workunit.client.0.trial106.stdout:mon.a: {} 2026-02-21T01:44:42.962 INFO:tasks.workunit.client.0.trial106.stdout:mon.b: {} 2026-02-21T01:44:42.970 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:194: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.1 2026-02-21T01:44:42.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:44:42.970 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:44:42.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:44:42.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:44:42.971 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:44:43.082 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: activate_osd td/ver-health 1 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/1 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:44:43.083 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-objectstore=memstore' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:44:43.084 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/1 2026-02-21T01:44:43.085 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-21T01:44:43.086 INFO:tasks.workunit.client.0.trial106.stdout:start osd.1 2026-02-21T01:44:43.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:44:43.087 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/1/whoami 2026-02-21T01:44:43.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-21T01:44:43.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:44:43.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:44:43.087 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:44:43.103 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:43.104+0000 7fc983440a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:43.104 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:43.104+0000 7fc983440a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:43.105 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:43.104+0000 7fc983440a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:44:43.107 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:43.108+0000 7fc983440a40 -1 Falling back to public interface 2026-02-21T01:44:43.116 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:44:43.116+0000 7fc983440a40 -1 osd.1 0 log_to_monitors true 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:43.191 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:44:43.192 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:44:43.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:43.192 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:43.301 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T01:44:44.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T01:44:44.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:44:44.302 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T01:44:44.302 INFO:tasks.workunit.client.0.trial106.stdout:1 2026-02-21T01:44:44.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:44:44.303 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T01:44:44.415 INFO:tasks.workunit.client.0.trial106.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 14 last_clean_interval [0,0) [v2:127.0.0.1:6808/754344662,v1:127.0.0.1:6809/754344662] [v2:127.0.0.1:6810/754344662,v1:127.0.0.1:6811/754344662] exists,up 171a9f42-8fa5-4002-8c09-11e5e3d63c3e 2026-02-21T01:44:44.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:44:44.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:44:44.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:44:44.416 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:200: TEST_check_version_health_3: sleep 10 2026-02-21T01:44:54.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: ceph health detail 2026-02-21T01:44:54.418 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2026-02-21T01:44:54.531 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:205: TEST_check_version_health_3: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 20 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:54.532 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:44:54.642 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:44:55.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:44:55.644 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:55.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:55.645 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:44:55.757 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:44:56.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:44:56.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:56.759 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:56.760 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:44:56.871 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:44:57.872 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:44:57.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:57.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:57.873 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:44:57.986 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:44:58.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:44:58.987 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:44:58.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:44:58.988 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:44:59.102 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:00.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:00.103 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:00.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:00.104 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:00.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:01.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:01.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:01.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:01.218 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:01.331 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:02.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:02.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:02.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:02.332 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:02.446 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:03.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:03.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:03.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:03.448 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:03.564 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:04.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:04.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:04.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:04.565 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:04.679 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:05.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:05.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:05.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:05.680 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:05.791 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:06.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:06.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:06.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:06.794 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:06.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-21T01:45:06.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-21T01:45:06.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-21T01:45:06.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-21T01:45:06.906 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-21T01:45:06.907 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:207: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:07.009 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2026-02-21T01:45:07.009 INFO:tasks.workunit.client.0.trial106.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2026-02-21T01:45:07.009 INFO:tasks.workunit.client.0.trial106.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2026-02-21T01:45:07.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:07.020 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2026-02-21T01:45:07.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:07.134 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2026-02-21T01:45:07.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:07.246 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2026-02-21T01:45:07.368 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:213: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.2 2026-02-21T01:45:07.369 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:45:07.369 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:45:07.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:45:07.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:45:07.370 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:45:07.480 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: activate_osd td/ver-health 2 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/2 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:45:07.481 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:45:07.482 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:45:07.483 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/2 2026-02-21T01:45:07.484 INFO:tasks.workunit.client.0.trial106.stdout:start osd.2 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/2/whoami 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:45:07.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:45:07.504 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.503+0000 7f315ce71a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:45:07.506 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.503+0000 7f315ce71a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:45:07.507 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.503+0000 7f315ce71a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:45:07.508 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.507+0000 7f315ce71a40 -1 Falling back to public interface 2026-02-21T01:45:07.516 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.515+0000 7f315ce71a40 -1 osd.2 0 log_to_monitors true 2026-02-21T01:45:07.590 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:45:07.591 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:45:07.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:45:07.592 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T01:45:07.703 INFO:tasks.workunit.client.0.trial106.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [0,0) [v2:127.0.0.1:6816/1968070036,v1:127.0.0.1:6817/1968070036] [v2:127.0.0.1:6818/1968070036,v1:127.0.0.1:6819/1968070036] exists,up fcc14b59-be2c-4ff2-b558-a349ec1d014a 2026-02-21T01:45:07.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:45:07.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:45:07.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:45:07.704 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:215: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.0 2026-02-21T01:45:07.704 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:45:07.705 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:45:07.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:45:07.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:45:07.705 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:45:07.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:45:07.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: ceph_debug_version_for_testing=02.00.00-gversion-test 2026-02-21T01:45:07.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: activate_osd td/ver-health 0 2026-02-21T01:45:07.814 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ver-health 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ver-health/0 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T01:45:07.815 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2026-02-21T01:45:07.816 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ver-health/0 2026-02-21T01:45:07.817 INFO:tasks.workunit.client.0.trial106.stdout:start osd.0 2026-02-21T01:45:07.818 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T01:45:07.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bafa2ddc-0fbe-44da-a6f6-b207b3c0046c --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.166630/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T01:45:07.819 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ver-health/0/whoami 2026-02-21T01:45:07.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T01:45:07.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T01:45:07.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T01:45:07.819 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T01:45:07.832 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.830+0000 7f94fafc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:45:07.834 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.830+0000 7f94fafc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:45:07.835 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.834+0000 7f94fafc4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T01:45:07.837 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.834+0000 7f94fafc4a40 -1 Falling back to public interface 2026-02-21T01:45:07.842 INFO:tasks.workunit.client.0.trial106.stderr:2026-02-21T01:45:07.838+0000 7f94fafc4a40 -1 osd.0 0 log_to_monitors true 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T01:45:07.926 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T01:45:07.927 INFO:tasks.workunit.client.0.trial106.stdout:0 2026-02-21T01:45:07.927 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T01:45:07.927 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T01:45:08.032 INFO:tasks.workunit.client.0.trial106.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [0,0) [v2:127.0.0.1:6800/1040665299,v1:127.0.0.1:6801/1040665299] [v2:127.0.0.1:6802/1040665299,v1:127.0.0.1:6803/1040665299] exists,up 1227070c-1332-4d12-b205-e20c86230d21 2026-02-21T01:45:08.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T01:45:08.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T01:45:08.033 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:218: TEST_check_version_health_3: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:08.034 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:08.140 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:09.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:09.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:09.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:09.142 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:09.256 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:10.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:10.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:10.257 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:10.258 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:10.372 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2026-02-21T01:45:11.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2026-02-21T01:45:11.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2026-02-21T01:45:11.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2026-02-21T01:45:11.374 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:11.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2026-02-21T01:45:11.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2026-02-21T01:45:11.485 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2026-02-21T01:45:11.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2026-02-21T01:45:11.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2026-02-21T01:45:11.486 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:220: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:11.604 INFO:tasks.workunit.client.0.trial106.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2026-02-21T01:45:11.604 INFO:tasks.workunit.client.0.trial106.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2026-02-21T01:45:11.604 INFO:tasks.workunit.client.0.trial106.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2026-02-21T01:45:11.604 INFO:tasks.workunit.client.0.trial106.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2026-02-21T01:45:11.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:11.616 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2026-02-21T01:45:11.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:11.732 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2026-02-21T01:45:11.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:11.842 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2026-02-21T01:45:11.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: ceph health detail 2026-02-21T01:45:11.973 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2026-02-21T01:45:12.090 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:45:12.091 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:45:12.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:45:12.200 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:45:12.200 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:45:12.201 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:45:12.201 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:45:12.202 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:45:12.202 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:45:12.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:45:12.203 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:45:12.204 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:45:12.204 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:45:12.205 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:45:12.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T01:45:12.206 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:45:12.214 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:45:12.214 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:45:12.214 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:45:12.215 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:45:12.216 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/ver-health 0 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ver-health 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ver-health KILL 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T01:45:12.217 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T01:45:12.219 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T01:45:12.219 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T01:45:12.220 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T01:45:12.220 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T01:45:12.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T01:45:12.221 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T01:45:12.222 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T01:45:12.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:45:12.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T01:45:12.223 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T01:45:12.224 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T01:45:12.224 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:45:12.225 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T01:45:12.226 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ver-health 2026-02-21T01:45:12.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T01:45:12.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T01:45:12.227 INFO:tasks.workunit.client.0.trial106.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.166630 2026-02-21T01:45:12.227 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.166630 2026-02-21T01:45:12.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T01:45:12.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T01:45:12.228 INFO:tasks.workunit.client.0.trial106.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-21T01:45:12.229 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T01:45:12.229 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T01:45:12.279 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2026-02-21T01:45:12.280 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-21T01:45:12.644 DEBUG:teuthology.parallel:result is None 2026-02-21T01:45:12.644 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-21T01:45:12.653 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-21T01:45:12.653 DEBUG:teuthology.orchestra.run.trial106:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-21T01:45:12.699 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-21T01:45:12.699 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-21T01:45:12.715 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-21T01:45:12.715 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-21T01:45:12.853 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-21T01:45:12.865 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-21T01:45:12.865 DEBUG:teuthology.orchestra.run.trial106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-21T01:45:13.293 INFO:teuthology.orchestra.run.trial106.stdout: remote refid st t when poll reach delay offset jitter 2026-02-21T01:45:13.293 INFO:teuthology.orchestra.run.trial106.stdout:============================================================================== 2026-02-21T01:45:13.293 INFO:teuthology.orchestra.run.trial106.stdout:*time1.weber.edu .PPS. 1 u 29 64 377 52.570 -5.346 2.994 2026-02-21T01:45:13.293 INFO:teuthology.orchestra.run.trial106.stdout:-163.123.153.14 162.159.200.123 4 u 32 64 377 69.268 -10.573 4.984 2026-02-21T01:45:13.293 INFO:teuthology.orchestra.run.trial106.stdout:+ntp.maxhost.io 132.163.97.4 2 u 14 64 357 69.671 -9.370 5.759 2026-02-21T01:45:13.293 INFO:teuthology.orchestra.run.trial106.stdout:+startkeylogger. 17.253.2.37 2 u 19 64 377 3.357 -8.758 3.734 2026-02-21T01:45:13.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-21T01:45:13.310 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-21T01:45:13.310 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-21T01:45:13.325 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-21T01:45:13.339 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-21T01:45:13.354 INFO:teuthology.task.internal:Duration was 2138.925661 seconds 2026-02-21T01:45:13.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-21T01:45:13.369 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-21T01:45:13.369 DEBUG:teuthology.orchestra.run.trial106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-21T01:45:13.396 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-21T01:45:13.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial106.front.sepia.ceph.com 2026-02-21T01:45:13.396 DEBUG:teuthology.orchestra.run.trial106:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-21T01:45:13.445 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-21T01:45:13.445 DEBUG:teuthology.orchestra.run.trial106:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-21T01:45:13.706 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-21T01:45:13.706 DEBUG:teuthology.orchestra.run.trial106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-21T01:45:13.717 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T01:45:13.718 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-21T01:45:13.718 INFO:teuthology.orchestra.run.trial106.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-21T01:45:13.718 INFO:teuthology.orchestra.run.trial106.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T01:45:13.718 INFO:teuthology.orchestra.run.trial106.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-21T01:45:13.787 INFO:teuthology.orchestra.run.trial106.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-21T01:45:13.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-21T01:45:13.809 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-21T01:45:13.809 DEBUG:teuthology.orchestra.run.trial106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-21T01:45:13.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-21T01:45:13.850 DEBUG:teuthology.orchestra.run.trial106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:45:13.886 INFO:teuthology.orchestra.run.trial106.stdout:kernel.core_pattern = core 2026-02-21T01:45:13.895 DEBUG:teuthology.orchestra.run.trial106:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-21T01:45:13.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T01:45:13.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-21T01:45:13.954 INFO:teuthology.task.internal:Transferring archived files... 2026-02-21T01:45:13.957 DEBUG:teuthology.misc:Transferring archived files from trial106:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-21_00:21:18-rados-wip-yuri3-testing-2026-02-19-1610-tentacle-distro-default-trial/62135/remote/trial106 2026-02-21T01:45:13.958 DEBUG:teuthology.orchestra.run.trial106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-21T01:45:14.014 INFO:teuthology.task.internal:Removing archive directory... 2026-02-21T01:45:14.014 DEBUG:teuthology.orchestra.run.trial106:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-21T01:45:14.047 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-21T01:45:14.059 INFO:teuthology.task.internal:Not uploading archives. 2026-02-21T01:45:14.059 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-21T01:45:14.084 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-21T01:45:14.085 DEBUG:teuthology.orchestra.run.trial106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-21T01:45:14.091 INFO:teuthology.orchestra.run.trial106.stdout: 1179659 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 01:45 /home/ubuntu/cephtest 2026-02-21T01:45:14.093 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-21T01:45:14.108 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-21T01:45:14.138 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 2138.925661087036 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-21T01:45:14.138 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-21T01:45:14.248 INFO:teuthology.run:pass